Topics Discussed:
-
-
- Overview
- Tools Description
- Securing Observability Tools
-
Overview
This topic outlines how to secure observability tools such as Prometheus and Grafana by employing SSL certificates and Nginx. Securing these tools is crucial for protecting data and maintaining privacy as they are pivotal for monitoring and visualising metrics.
Importance of Security:
- Prometheus and Grafana can be accessed via public IP and port numbers.
- Unencrypted communication exposes application details.
- Securing communication with SSL and enhancing security using Nginx is essential.
Tools Description
Grafana
- Purpose: Visualizes logs and metrics.
- Access: Installed on infrastructure, accessed via IP or URL.
- Default Port: 3000.
Prometheus
- Purpose: Gathers and stores metrics; commonly used as a data source for Grafana.
- Access: Installed on infrastructure, accessed via IP or URL.
- Default Port: 9090.
Nginx
- Purpose: Web server with load balancing and reverse proxy features.
- Role in Security: Handles requests, performs reverse proxy functions, and manages configurations.
SSL (Secure Socket Layer)
- Purpose: Encrypts data to secure connections.
- Function: Converts an unsecured connection into a secured one by encrypting data transmitted over the network.
Securing Observability Tools
1. Introduction to the Setup
Current Scenario:
- Grafana and Prometheus are accessible via their respective ports (3000 and 9090).
- Direct access exposes the applications to potential security risks.
Solution:
Implement SSL certificates and use Nginx as a reverse proxy to secure and manage traffic.
2. Using Nginx as a Reverse Proxy
Role of Nginx:
- Handles Incoming Requests: Manages traffic by routing requests to the appropriate application.
- Reverse Proxy Functionality: Routes requests to Prometheus and Grafana using local loopback addresses (127.0.0.1:3000 and 127.0.0.1:9090).
Benefits:
- Private Communication: Requests between Nginx and the applications remain internal, reducing exposure.
- Enhanced Security: By routing traffic through Nginx, we minimise direct public access to Grafana and Prometheus.
3. Implementing SSL Certificates
Purpose of SSL:
- Encrypt Communication: Secures data transmitted between users and the Nginx proxy server.
- Authenticate Connections: Ensures that connections are made to a trusted source.
Steps to Implement SSL:
- Obtain SSL Certificates: Acquire certificates from a trusted Certificate Authority (CA).
- Configure Nginx: Set up Nginx to use the SSL certificates for securing connections.
- Update Configuration: Adjust Nginx settings to handle HTTPS traffic and forward it to Grafana and Prometheus securely.
Outcome:
This setup ensures that both external and internal traffic is encrypted and that Grafana and Prometheus are not directly exposed to public networks.
Configuration commands
- Create an ubuntu instance
- From the terminal install Docker, Prometheus, Grafana and Nginxapt update
- Install Docker engine
apt install docker.io
- Install Prometheus and Grafana
docker run -d –name prometheus -p 9090:9090 prom/prometheus
docker run -d – name grafana -p 3000:3000 grafana/grafana
docker ps
Note: Now Prometheus and Grafana will be accessible using the host public IP and port number of the application
- Install nginx
apt install nginx -y
- Install socat for internal communication
apt install socat -y
- Create and point domain of prometheus and Grafana, here we use https://www.duckdns.org/ for creating domain name
promm.duckdns.org
graffaa.duckdns.org
- Generate ssl certificate
clone acme(Certificate management tool) from github
git clone https://github.com/acmesh-official/acme.sh
cd acme.sh
./acme.sh – -issue -d promm.duckdns.org -w /var/www/html -m <mail id> – -server zerossl
./acme.sh – -issue -d graffaa.duckdns.org -w /var/www/html -m <mail id> – -server zerossl
Note: Save the certificate path of both prometheus and Grafana to configure in Nginx configuration file
- Nginx configuration
cd /etc/nginx
cd sites-available
configuration file for prometheus
server {
listen 443 ssl;
server_name promm.duckdns.org ;
ssl_certificate /root/.acme.sh/promm.duckdns.org_ecc/fullchain.cer;
ssl_certificate_key /root/.acme.sh/promm.duckdns.org_ecc/promm.duckdns.org.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://127.0.0.1:9090;
}
}
configuration file for grafana
server {
listen 443 ssl;
server_name graffa.duckdns.org ;
ssl_certificate /root/.acme.sh/graffa.duckdns.org_ecc/fullchain.cer;
ssl_certificate_key /root/.acme.sh/graffa.duckdns.org_ecc/graffa.duckdns.org.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://127.0.0.1:3000;
}
}
- Enable soft link from site-available to site-enabled
ln -s /etc/nginx/sites-available/prom /etc/nginx/sites-enabled
ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled
- Test Nginx configuration
nginx -t
- Apply settings by reloading nginx
systemctl reload nginx
Access Prometheus using secure link https://promm.duckdns.org
Access Grafana using secure link https://graffa.duckdns.org
