Prometheus Metrics

Ditto exposes an endpoint at /metrics that returns data in Prometheus exposition format (aka OpenMetrics format). This data can be scraped by Prometheus to create real-time graphs of Ditto.

INFO

Prometheus was created by SoundCloud in 2012. It became so ubiquitous that OpenMetrics sought to standardize its custom data format.

Installing Prometheus

1. Add the user for Prometheus

For security reasons, it is always advised to run services as a non-root user. Run the following command to create a new user without a home directory:

useradd --no-create-home --shell /bin/false prometheus

2. Download Prometheus

Visit the official Prometheus download page (https://prometheus.io/download/) to get the link to the latest version of Prometheus. You can use the wget command to download it. As an example, here's how to download Prometheus 2.53.0:

wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz

3. Extract the downloaded file

Use the tar command to extract the downloaded file:

tar -xvf prometheus-2.53.0.linux-amd64.tar.gz

4. Move the configuration files and set the owner

We'll move the necessary files to their required directories:

mv prometheus-2.31.1.linux-amd64/prometheus /usr/local/bin/
mv prometheus-2.31.1.linux-amd64/promtool /usr/local/bin/
mkdir /etc/prometheus
mkdir /var/lib/prometheus
mv prometheus-2.31.1.linux-amd64/consoles /etc/prometheus
mv prometheus-2.31.1.linux-amd64/console_libraries /etc/prometheus
mv prometheus-2.31.1.linux-amd64/prometheus.yml /etc/prometheus

Then, change the ownership of these files to the prometheus user:

chown prometheus:prometheus /usr/local/bin/prometheus
chown prometheus:prometheus /usr/local/bin/promtool
chown -R prometheus:prometheus /etc/prometheus/
chown -R prometheus:prometheus /var/lib/prometheus/

5. Create a service file

Now, we'll create a system service file for Prometheus. Open a new service file in your text editor:

nano /etc/systemd/system/prometheus.service

Paste the following content into it:

[Unit]
Description=Prometheus Service
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Save and close the file.

6. Reload systemd and start Prometheus

Reload the systemd system manager configuration. Then, start the Prometheus service and enable it to launch everytime at system boot:

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

7. Check the status of Prometheus

Verify that Prometheus is running correctly:

systemctl status prometheus

You should see an output saying that the service is active (running).

Now you can access Prometheus by starting a new ssh tunnel to your server:

ssh -L 9090:localhost:9090 user@your_server_ip

Then, open your browser and navigate to http://localhost:9090.

Useful Queries

Here are some useful queries that you can run in Prometheus:

Available Database Connections

db_available_connections

Active Relay Connections

relay_connections

Rate of Relay Messages by Verb

rate(relay_messages_total[5m])

Rate of Relay Events by Kind

rate(relay_events_total[5m])

Rate of Firehose Events by Kind

rate(firehose_events_total[5m])

Rate of Database Events by Kind

rate(db_events_total[5m])

Rate of Fetch Requests

rate(fetch_total[5m])