How to Use Prometheus and Grafana for Server Monitoring -

Prometheus and Grafana are popular open-source tools for monitoring and visualizing server and application performance. Prometheus collects and stores metrics, while Grafana provides a beautiful dashboard for visualization. Here’s a guide to setting up server monitoring using Prometheus and Grafana on a Linux server.

Step 1: Install Prometheus #

Download Prometheus:
- Visit the Prometheus downloads page and copy the link for the latest release for your OS, or run:

wget https://github.com/prometheus/prometheus/releases/download/v2.40.0/prometheus-2.40.0.linux-amd64.tar.gz

2. Extract the Prometheus package:

tar -xvf prometheus-2.40.0.linux-amd64.tar.gz
cd prometheus-2.40.0.linux-amd64

3. Move Prometheus binaries to /usr/local/bin:

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/

4. Set up Prometheus configuration:

Create a directory for Prometheus configuration:

sudo mkdir /etc/prometheus
sudo mv prometheus.yml /etc/prometheus/

Edit prometheus.yml to define targets:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

5. Create a Prometheus systemd service file:

sudo nano /etc/systemd/system/prometheus.service

Add the following:

[Unit]
Description=Prometheus Service
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/

[Install]
WantedBy=multi-user.target

6. Start and enable Prometheus:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

7. Verify Prometheus is running by navigating to http://<your_server_ip>:9090.

Step 2: Install and Configure Node Exporter #

Node Exporter collects metrics from the server, such as CPU, memory, and disk usage.

Download Node Exporter:

wget https://github.com/prometheus/node_exporter/releases/download/v1.4.0/node_exporter-1.4.0.linux-amd64.tar.gz

2. Extract Node Exporter:

tar -xvf node_exporter-1.4.0.linux-amd64.tar.gz
sudo mv node_exporter-1.4.0.linux-amd64/node_exporter /usr/local/bin/

3. Create a Node Exporter systemd service file:

sudo vi /etc/systemd/system/node_exporter.service

Add the following:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

4. Start and enable Node Exporter:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

5. Configure Prometheus to scrape Node Exporter metrics:

Edit /etc/prometheus/prometheus.yml to add Node Exporter as a target:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

6. Restart Prometheus:

sudo systemctl restart prometheus

Step 3: Install Grafana #

Add the Grafana APT repository:

sudo apt-get install -y software-properties-common

sudo apt-add-repository "deb https://packages.grafana.com/oss/deb stable main"

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

2. Install Grafana:

sudo apt-get update
sudo apt-get install grafana -y

3. Start and enable Grafana:

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

4. Access Grafana:

Open your browser and navigate to http://<your_server_ip>:3000. The default login is admin for both the username and password (you’ll be prompted to change the password after logging in).

Step 4: Configure Grafana to Use Prometheus as a Data Source #

Add Prometheus as a data source:
- In the Grafana dashboard, go to Configuration > Data Sources.
- Select Add data source and choose Prometheus.
- Set the URL to http://localhost:9090 and click Save & Test to ensure it’s connected.
Create a Dashboard:
- Go to + > Dashboard to create a new dashboard.
- Select Add New Panel and choose metrics you want to visualize from Prometheus.

Step 5: Create Alerts (Optional) #

Grafana and Prometheus support alerting based on metric conditions. Here’s a basic setup for server load alerts:

Configure Prometheus alert rules:

Create a new rules file for Prometheus

sudo vi /etc/prometheus/alert.rules

Define an alert rule (e.g., high CPU usage):

groups:
  - name: example_alert
    rules:
      - alert: HighCPUUsage
        expr: node_cpu_seconds_total{mode="idle"} < 20
        for: 5m
        labels:
          severity: "critical"
        annotations:
          summary: "High CPU Usage Detected"
          description: "CPU usage has been above 80% for the last 5 minutes."

2. Update Prometheus configuration to load the alert rules:

Edit /etc/prometheus/prometheus.yml and add:

rule_files:
  - "/etc/prometheus/alert.rules"

3. Restart Prometheus:

sudo systemctl restart prometheus

4. Set up Grafana Alerting (for more advanced alerting):

In Grafana, go to Alerting > Notification channels to set up notification channels (e.g., email or Slack).
Define alert conditions in your Grafana dashboards, setting thresholds to trigger notifications.

Step 6: Monitor and Visualize #

With Prometheus and Grafana set up:

Check Prometheus at http://<your_server_ip>:9090 to ensure metrics are being collected.
View Grafana Dashboards at http://<your_server_ip>:3000 to see your data visualizations.

Author’s Final Word #

By following these steps, you’ll have a fully functioning monitoring solution for your server. Prometheus and Grafana will gather and display metrics, helping you keep track of your server’s performance and set up alerts for proactive monitoring.