Prometheus is a flexible monitoring solution in development since 2012. The software stores all its data in a time series database and offers a multi-dimensional data model and a powerful query language to generate reports of the monitored resources.
This tutorial makes no assumptions about previous knowledge, other than:
You are comfortable with a Linux operating system, specifically Ubuntu 20.04
You are comfortable being able to ssh into your node, as all operations will be done from the command line\
As your Prometheus is only capable of collecting metrics, we want to extend its capabilities by adding Node Exporter, a tool that collects information about the system including CPU, disk, and memory usage and exposes them for scraping.
Download the latest version of Node Exporter:
Unpack the downloaded archive. This will create a directory node_exporter-1.2.2.linux-amd64, containing the executable, a readme and license file:
Copy the binary file into the directory /usr/local/bin and set the ownership to the user you have created in step previously:
Remove the leftover files of Node Exporter, as they are not needed any longer:
To run Node Exporter automatically on each boot, a Systemd service file is required. Create the following file by opening it in Nano:
Copy the following information in the service file, save it and exit Nano:
Collectors are used to gather information about the system. By default a set of collectors is activated. You can see the details about the set in the README-file. If you want to use a specific set of collectors, you can define them in the ExecStart section of the service. Collectors are enabled by providing a --collector.<name> flag. Collectors that are enabled by default can be disabled by providing a --no-collector.<name> flag.
Reload Systemd to use the newly defined service:
Run Node Exporter by typing the following command:
Verify that the software has been started successfully:
You will see an output like this, showing you the status active (running) as well as the main PID of the application:
If everything is working, enable Node Exporter to be started on each boot of the server:
You will need to create new users for running Prometheus securely. This can be done by doing:
Create the directories for storing the Prometheus binaries and its config files:
Set the ownership of these directories to our prometheus user, to make sure that Prometheus can access to these folders:
Prior to using Prometheus, it needs basic configuring. Thus, we need to create a configuration file named prometheus.yml
The configuration file of Prometheus is written in YAML which strictly forbids to use tabs. If your file is incorrectly formatted, Prometheus will not start. Be careful when you edit it.
Open the file prometheus.yml in a text editor:
Prometheus’ configuration file is divided into three parts: global, rule_files, and scrape_configs.
In the global part we can find the general configuration of Prometheus: scrape_interval defines how often Prometheus scrapes targets, evaluation_interval controls how often the software will evaluate rules. Rules are used to create new time series and for the generation of alerts.
The rule_files block contains information of the location of any rules we want the Prometheus server to load.
The last block of the configuration file is named scape_configs and contains the information which resources Prometheus monitors.
Our file should look like this example:
The global scrape_interval is set to 15 seconds which is enough for most use cases.
We do not have any rule_files yet, so the lines are commented out and start with a #.
In the scrape_configs part we have defined our first exporter. It is Prometheus that monitors itself. As we want to have more precise information about the state of our Prometheus server we reduced the scrape_interval to 5 seconds for this job. The parameters static_configsand targets determine where the exporters are running. In our case it is the same server, so we use localhost and the port 9090.
As Prometheus scrapes only exporters that are defined in the scrape_configs part of the configuration file, we have to add Node Exporter to the file, as we did for Prometheus itself.
We add the following part below the configuration for scraping Prometheus:
Overwrite the global scrape interval again and set it to 5 seconds. As we are scarping the data from the same server as Prometheus is running on, we can use localhost with the default port of Node Exporter: 9100.
If you want to scrape data from a remote host, you have to replace localhost with the IP address of the remote server.
Tip: For all information about the configuration of Prometheus, you may check the configuration documentation.
Set the ownership of the file to our Prometheus user:
Our Prometheus server is ready to run for the first time.
Start Prometheus directly from the command line with the following command, which executes the binary file as our Prometheus user:
The server starts displaying multiple status messages and the information that the server has started:
Open your browser and type http://IP.OF.YOUR.SERVER:9090 to access the Prometheus interface. If everything is working, we end the task by pressing on CTRL + C on our keyboard.
If you get an error message when you start the server, double-check your configuration file for possible YAML syntax errors. The error message will tell you what to check.
The server is working now, but it cannot yet be launched automatically at boot. To achieve this, we have to create a new systemd configuration file that will tell your OS which services should it launch automatically during the boot process.
The service file tells systemd to run Prometheus as prometheus and specifies the path of the configuration files.
Copy the following information in the file and save it, then exit the editor:
To use the new service, reload systemd:
We enable the service so that it will be loaded automatically during boot:
Start Prometheus:
Your Prometheus server is ready to be used.
We have now installed Prometheus to monitor your instance. Prometheus provides a basic web server running on http://your.server.ip:9000 that provide access to the data collected by the software.
Download and Unpack Prometheus latest release of Prometheus:
The following two binaries are in the directory:
Prometheus - Prometheus main binary file
Promtool
The following two folders (which contain the web interface, configuration files examples and the license) are in the directory:
Consoles
Console_libraries
Copy the binary files into the /usr/local/bin/directory:
Set the ownership of these files to the prometheus user previously created:
Copy the consoles and console_libraries directories to /etc/prometheus:
Set the ownership of the two folders, as well as of all files that they contain, to our prometheus user:
In our home folder, remove the source files that are not needed anymore: