Prometheus is a flexible monitoring solution in development since 2012. The software stores all its data in a time series database and offers a multi-dimensional data model and a powerful query language to generate reports of the monitored resources.

This tutorial makes no assumptions about previous knowledge, other than:

1. You are comfortable with a Linux operating system, specifically Ubuntu 20.04

2. You are comfortable being able to ssh into your node, as all operations will be done from the command line\

Monitoring is inmensely important so to ensure the liveness and reliabilty of your infrastructure. If your validator is not signing blocks it will eventually get slashed losing you and your delegators some of their SCRT balance. Same for full nodes it is important they are able to serve queries as if they are down performance of dApps and other applications will be limited.\

Monitoring is best done by a dedicated piece of software that provides both analytics and alerts. Some of those options are laid out below so to help you set them up. Consider relying on more than 1 monitoring solutions and leverage external RPCs so to secure your setup even further.

Overview

• Prometheus

• Grafana

• Docker

• PagerDuty

• Goaccess

As your Prometheus is only capable of collecting metrics, we want to extend its capabilities by adding Node Exporter, a tool that collects information about the system including CPU, disk, and memory usage and exposes them for scraping.

Download the latest version of Node Exporter:

wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz

Unpack the downloaded archive. This will create a directory node_exporter-1.2.2.linux-amd64, containing the executable, a readme and license file:

tar xvf node_exporter-1.2.2.linux-amd64.tar.gz

Copy the binary file into the directory /usr/local/bin and set the ownership to the user you have created in step previously:

sudo cp node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Remove the leftover files of Node Exporter, as they are not needed any longer:

rm -rf node_exporter-0.16.0.linux-amd64.tar.gz node_exporter-0.16.0.linux-amd64

To run Node Exporter automatically on each boot, a Systemd service file is required. Create the following file by opening it in Nano:

sudo nano /etc/systemd/system/node_exporter.service

Copy the following information in the service file, save it and exit Nano:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Collectors are used to gather information about the system. By default a set of collectors is activated. You can see the details about the set in the README-file. If you want to use a specific set of collectors, you can define them in the ExecStart section of the service. Collectors are enabled by providing a --collector.<name> flag. Collectors that are enabled by default can be disabled by providing a --no-collector.<name> flag.

Reload Systemd to use the newly defined service:

sudo systemctl daemon-reload

Run Node Exporter by typing the following command:

sudo systemctl start node_exporter

Verify that the software has been started successfully:

sudo systemctl status node_exporter

You will see an output like this, showing you the status active (running) as well as the main PID of the application:

● node_exporter.service - Node Exporter
Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2018-06-25 11:47:06 UTC; 4s ago
Main PID: 1719 (node_exporter)
CGroup: /system.slice/node_exporter.service
└─1719 /usr/local/bin/node_exporter

If everything is working, enable Node Exporter to be started on each boot of the server:

sudo systemctl enable node_exporter

Install Grafana on our instance which queries our Prometheus server.

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_8.1.4_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_8.1.4_amd64.deb

Enable the automatic start of Grafana by systemd:

sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server

Grafana is running now, and we can connect to it at http://your.server.ip:3000. The default user and password is admin / admin.

Now you have to create a Prometheus data source: - Click the Grafana logo to open the sidebar. - Click “Data Sources” in the sidebar. - Choose “Add New”. - Select “Prometheus” as the data source. - Set the Prometheus server URL (in our case: http://localhost:9090/) - Click “Add” to test the connection and to save the new data source.

Grafana allows you to easily visualize your monitoring results and other analytics

Overview

From here, you're going to want to set up alerts for if something happens with your node, which will be a follow-up document.

Docker and Docker Compose will allow you to run the required monitoring applications with a few commands. These instructions will run the following:

• Grafana on port 3000: An open source interactive analytics dashboard.

• Prometheus on port 9090: An open source metric collector.

• Node Exporter on port 9100: An open source hardware metric exporter.

Install Grafana on our instance which queries our Prometheus server.

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_8.1.4_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_8.1.4_amd64.deb

Enable the automatic start of Grafana by systemd:

sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server

Grafana is running now, and we can connect to it at http://your.server.ip:3000. The default user and password is admin / admin.

Now you have to create a Prometheus data source:

• Click the Grafana logo to open the sidebar.

• Click “Data Sources” in the sidebar.

• Choose “Add New”.

• Select “Prometheus” as the data source

• Set the Prometheus server URL (in our case: http://localhost:9090/)

• Click “Add” to test the connection and to save the new data source

Installing Cosmos SDK Grafana Dashboard

Finally, we're going to install a basic dashboard for Cosmos SDKs. For further reference in these steps, see: https://github.com/zhangyelong/cosmos-dashboard

Enable Tendermint Metrics

sed -i 's/prometheus = false/prometheus = true/g' <YOUR-NODE-HOMEDIR>/config/config.toml

After restarting your node, you should be able to access the tendermint metrics(default port is 26660): http://localhost:26660

Configure Prometheus Targets

Append a job under the scrape_configs of your prometheus.yml

  - job_name: secret-network
	static_configs:
	- targets: ['<Validator-IP>:26660']
	  labels:
		instance: validator
	- targets: ['<Sentry-0-IP>:26660']
	  labels:
		instance: sentry-0
	- targets: ['<Sentry-1-IP>:26660']
	  labels:
		instance: sentry-1

Reload Prometheus Config

curl -X POST http://<PROMETHEUS-IP>:9090/-/reload

Import Grafana Dashboard

Copy and paste the Grafana Dashboard ID 11036 OR content of cosmos-dashboard.json, click on Load to complete importing.

1. Set chain-id to secret-3

2. You're done!\

Download and Unpack Prometheus latest release of Prometheus:

sudo apt-get update && apt-get upgrade
wget https://github.com/prometheus/prometheus/releases/download/v2.30.0/prometheus-2.30.0.linux-amd64.tar.gztar xfz prometheus-_.tar.gzcd prometheus-_

The following two binaries are in the directory:

• Prometheus - Prometheus main binary file

• Promtool

The following two folders (which contain the web interface, configuration files examples and the license) are in the directory:

• Consoles

• Console_libraries

Copy the binary files into the /usr/local/bin/directory:

sudo cp ./prometheus /usr/local/bin/
sudo cp ./promtool /usr/local/bin/

Set the ownership of these files to the prometheus user previously created:

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy the consoles and console_libraries directories to /etc/prometheus:

sudo cp -r ./consoles /etc/prometheus
sudo cp -r ./console_libraries /etc/prometheus

Set the ownership of the two folders, as well as of all files that they contain, to our prometheus user:

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

In our home folder, remove the source files that are not needed anymore:

cd .. && rm -rf prometheus-\*

The docker images expose the following ports:

• 3000 Grafana. Your main dashboard. Default login is admin\admin.

• 9090 Prometheus. Access to this port should be restricted.

• 9100 Node Exporter. Access to this port should be restricted.

• Your secret node metrics on port 26660 should also be restricted.

If you followed the basic security guide, these ports are already restricted. You will need to allow the grafana port:

sudo ufw allow 3000

You can also allow access from a specific IP if desired:

sudo ufw allow from 123.123.123.123 to any port 3000

Clone the and descend into the monitoring folder:

In the Prometheus folder, modify cosmos.yaml, replace NODE_IP with the IP of your node. (If your node is on the docker host machine, use 172.17.0.1)

Replace the default Prometheus config with the modified cosmos.yaml

Preparing Your Environment

• You will need to install docker and docker-compose.

• The following instructions assume Ubuntu 20.04 on an x86-64 CPU.

Install Docker

Test the installation:

Apply executable permissions to the binary:

Download the current stable release of Docker Compose:

Install Docker Compose

Test the installation:

Install docker:

Setup the docker stable repository:

Add Docker’s official GPG key:

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

The dashboard for Cosmos SDK nodes is pre-installed, to use it:

Enable Tendermint metrics in your secret-node

After restarting your node, you should be able to access the Tendermint metrics (default port is 26660):

If you did not replace NODE_IP with the IP of your node in the Prometheus config, do so now. If your node is on the docker host machine, use 172.17.0.1

1. Login to Grafana and open the Cosmos Dashboard from the page.

2. Set the chain-id to secret-3

Start the containers deploying the monitoring stack (Grafana + Prometheus + Node Exporter):

Login to Grafana at http://your-ip:3000 with username admin and password admin

The containers will restart automatically after rebooting unless they are stopped manually.

From the node_tooling/monitoring directory:

Editing Nginx Configuration

Configure Nginx to format logs and set up a server block.

1. Open the Nginx configuration file:

sudo nano /etc/nginx/nginx.conf 

1. Add the following log format into your http group in nginx:

http {  
    log_format combined_log '$remote_addr - $remote_user [$time_local] '
                   '"$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent"';
    # ... (rest of the configuration)
}

1. (optional) Instead anonymize IP addresses in logs:

http {
    map $remote_addr $anonymized_addr {
                ~(?P<ip>\d+\.\d+)\.\d+\.    $ip.0.0;
                ~(?P<ip>[^:]+:[^:]+):       $ip::;
                # IP addresses to not anonymize (such as your server)
                127.0.0.1                   $remote_addr;
                ::1                         $remote_addr;
                #w.x.y.z                    $remote_addr;
                #a::c:d::e:f                $remote_addr;
                default                     0.0.0.0;
    }
    
    log_format combined_log '$anonymized_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent"';
                       
    access_log  /var/log/nginx/access.log  combined_log;
    
    # ... (rest of the configuration)
}

1. Configure a server block:

server {
    listen 80;
    server_name yourdomain.com;  # Replace with your domain

    root /var/www/html;
    index report.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Testing and Reloading Nginx Configuration

1. Test the new configuration:

sudo nginx -t

1. Reload Nginx to apply changes:

sudo systemctl reload nginx

Configuring Log Rotation

Log rotation in Nginx is a process for managing log files to prevent them from becoming excessively large and consuming too much disk space. As Nginx continuously logs web requests, these files can grow rapidly. Without rotation, they can lead to performance issues and make log analysis more difficult. The default setting is for log rotation is daily, which means that the logs that goaccess can use for its reporting are also only daily. To increase that timeframe, do the following:

1. Edit log rotation configuration:

sudo nano /etc/logrotate.d/nginx

1. Add the configuration, please change the monthly to daily or weekly if you need daily or weekly rotation of the logs.

/var/log/nginx/*.log {
        monthly
        missingok
        rotate 1
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

1. Apply the new rotation configuration:

sudo logrotate -f /etc/logrotate.d/nginx

Setting Up GoAccess for Web Traffic Monitoring

1. Generate a HTML report:

sudo goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED

If you wish to automate this, use crontab to generate recurring reports:

1. Open crontab for editing (use sudo, otherwise crontab will not access to the log file):

sudo crontab -e

1. Add the line to automate hourly report generation:

0 * * * * goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED

Before configuring Nginx, install GoAccess, a real-time web log analyzer.

1. Update your package lists:

sudo apt update

1. Install GoAccess:

sudo apt install goaccess

Goaccess is a powerful tool when it comes to providing usage statistics for your endpoints.

This tutorial will guide you through configuring Nginx for logging, anonymizing logs, monitoring web traffic with GoAccess, and setting up log rotation for Nginx logs.

This guide is intended for intermediate users who are familiar with Linux, Nginx, and using the command-line interface.

1. Install Goaccess

2. Setup Goaccess

Creating Users

You will need to create new users for running Prometheus securely. This can be done by doing:

sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus 
sudo useradd --no-create-home --shell /bin/false node_exporter

Creating Directories

1. Create the directories for storing the Prometheus binaries and its config files:

sudo mkdir /etc/prometheus 
sudo mkdir /var/lib/prometheus

Set Directory Ownership

1. Set the ownership of these directories to our prometheus user, to make sure that Prometheus can access to these folders:

sudo chown prometheus:prometheus /etc/prometheus 
sudo chown prometheus:prometheus /var/lib/prometheus

Prior to using Prometheus, it needs basic configuring. Thus, we need to create a configuration file named prometheus.yml

Open the file prometheus.yml in a text editor:

Prometheus’ configuration file is divided into three parts: global, rule_files, and scrape_configs.

In the global part we can find the general configuration of Prometheus: scrape_interval defines how often Prometheus scrapes targets, evaluation_interval controls how often the software will evaluate rules. Rules are used to create new time series and for the generation of alerts.

The rule_files block contains information of the location of any rules we want the Prometheus server to load.

The last block of the configuration file is named scape_configs and contains the information which resources Prometheus monitors.

Our file should look like this example:

The global scrape_interval is set to 15 seconds which is enough for most use cases.

We do not have any rule_files yet, so the lines are commented out and start with a #.

In the scrape_configs part we have defined our first exporter. It is Prometheus that monitors itself. As we want to have more precise information about the state of our Prometheus server we reduced the scrape_interval to 5 seconds for this job. The parameters static_configsand targets determine where the exporters are running. In our case it is the same server, so we use localhost and the port 9090.

As Prometheus scrapes only exporters that are defined in the scrape_configs part of the configuration file, we have to add Node Exporter to the file, as we did for Prometheus itself.

We add the following part below the configuration for scraping Prometheus:

Overwrite the global scrape interval again and set it to 5 seconds. As we are scarping the data from the same server as Prometheus is running on, we can use localhost with the default port of Node Exporter: 9100.

If you want to scrape data from a remote host, you have to replace localhost with the IP address of the remote server.

Set the ownership of the file to our Prometheus user:

Our Prometheus server is ready to run for the first time.

Running Prometheus

1. Start Prometheus directly from the command line with the following command, which executes the binary file as our Prometheus user:

The server starts displaying multiple status messages and the information that the server has started:

Open your browser and type http://IP.OF.YOUR.SERVER:9090 to access the Prometheus interface. If everything is working, we end the task by pressing on CTRL + C on our keyboard.

The server is working now, but it cannot yet be launched automatically at boot. To achieve this, we have to create a new systemd configuration file that will tell your OS which services should it launch automatically during the boot process.

The service file tells systemd to run Prometheus as prometheus and specifies the path of the configuration files.

Copy the following information in the file and save it, then exit the editor:

To use the new service, reload systemd:

We enable the service so that it will be loaded automatically during boot:

Start Prometheus:

Your Prometheus server is ready to be used.

We have now installed Prometheus to monitor your instance. Prometheus provides a basic web server running on http://your.server.ip:9000 that provide access to the data collected by the software.