Example of Monitoring a Server Room on the Basis of Icinga and NetPing Devices

  • Published In: Tutorial
  • Created Date: 2019-01-11
  • Hits: 6050

Icinga is a monitoring system with the open source code. It was originally created as a branch of the Nagios monitoring system and resembles it very much. In this article, we will examine the example of configuring Icinga for working with the server room microclimate monitoring device UniPing server solution v4/SMS and install add-ons for building graphs and a map.

To implement a regarded solution, we will need:

  1. A server with the installed Debian 9 (in the article, the IP address is 192.168.0.45);
  2. UniPing server solution v4/SMS (in the article, the IP address is 192.168.0.100);
  3. 1-Wire temperature sensor;
  4. 1-Wire humidity sensor;
  5. Door sensor;
  6. Liquid sensor;
  7. Airflow sensor;
  8. Supply voltage sensor

Configuring a Monitoring Device UniPing server solution v4/SMS for Working with the Selected Set of Sensors

To start the work with a device, there is a need to perform its basic configuration. To do this, go to the settings section of a device (1), configure necessary network parameters (2) and access parameters (3). 

Basic configuration of UniPing server solution v4SMS

To connect a 1-Wire sensor, we need to know its unique ID. The connection process is described in this article very well. The information on connecting and configuring sensors can be obtained from the official documentation.

1-Wire Temperature Sensors

Configuring temperature sensors UniPing server solution v4SMS

1-Wire Humidity Sensors

Configuring humidity sensors UniPing server solution v4SMS

Such sensors as a supply voltage sensor, a liquid sensor, a door sensor, and an airflow sensor are connected to IO lines. These sensors are dry contact type sensors. In the documentation, there is a detailed description of configuringinput-output lines (IO lines).

Configuring sensors of dry contact type UniPing server solution v4SMS

Installing Icinga

Installing a monitoring system is sufficiently well described in the official documentation and is not difficult. Let's point out only the basic moments of the installation:

All commands in the console that are represented hereinafter, have been run as root

First, add a repository:

 

apt-get -y install apt-transport-https

wget -O - https://packages.icinga.com/icinga.key | apt-key add -
echo 'deb https://packages.icinga.com/debian icinga-stretch main' >/etc/apt/sources.list.d/icinga.list
apt-get update

And install Icinga:

 

apt-get install icinga2

Then, there is a need to install a basic set of plugins for monitoring (that are going to be located in /usr/lib/nagios/plugins later):

 

apt-get install monitoring-plugins

Then, restart Icinga and check its status:

 

systemctl restart icinga2

systemctl status icinga2

At this point, the installation of Icinga is completed. The system is installed and operates fine.

Installing and Configuring the Database Icinga Data Output (DB IDO)

IDO is designed for entering all the data from the monitoring system to the database. Later the process will be illustrated on the example of the MySQL database, the variant of PostgreSQL is also possible, it is regarded in details in the official documentation.

First, there is a need to install necessary packages:

 

apt-get install mysql-server mysql-client

mysql_secure_installation

Then, the IDO module for Icinga:

 

apt-get install icinga2-ido-mysql

Next, we need to configure IDO. To do this, create a database:

 

# mysql -u root -p

CREATE DATABASE icinga;
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga.* TO 'icinga'@'localhost' IDENTIFIED BY 'icinga';
quit

Then, import the structure:

 

mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/mysql.sql

Switch on IDO and restart Icinga:

 

# icinga2 feature enable ido-mysql

systemctl restart icinga2

A configuration file is located at /etc/icinga2/features-available/ido-mysql.conf. If necessary, you should change the data in it for connecting it to the base.

Then, there is a need to configure REST API to connect Icinga and Icinga Web 2:

 

icinga2 api setup

And restart Icinga again so that the changes take effect:

 

systemctl restart icinga2

Installing Icinga Web 2

Icinga Web 2 is a PHP framework that is an intuitive user interface for the detailed supervision over the big number of services. In addition, it allows controlling monitoring processes by sending external commands to Icinga in one click. Its installation, the same as the one of Icinga, is satisfactorily described in the official documentation. We will point out some basic moments:

 

apt-get install icingaweb2 icingacli

Then, there is a need to create a token for authorizing Icinga Web 2 connection to Icinga:

 

icingacli setup token create

After, prepare a database:

 

mysql -u root -p

MariaDB [mysql]> CREATE DATABASE icingaweb2;

MariaDB [mysql]> GRANT ALL ON icingaweb2.* TO icingaweb2@localhost IDENTIFIED BY 'icingapasswd';

Further configuring will be performed in the web-интерфейсе Icinga Web 2. To do this, go to http://192.168.0.45/icinga/setup. We are welcomed by the window with a request to enter the token obtained before:

Configuring Icinga Web 2

Then, we are asked to choose the necessary modules. We are interested in the monitoring module only, so select it and continue.

Configuring Icinga Web 2

We will see the list of requirements to the system. If not all of them are marked green, there is a need to go back to the console and update/install a corresponding package.

Configuring Icinga Web 2

Then, choose the authentication type Database.

Configuring Icinga Web 2

The next necessary step is configuring parameters for connecting to the database:

  1. Resource Name: icingaweb2 - is the name of the data resource;
  2. Database Type: MySQL - in this article, we use the MySQL base;
  3. Host: localhost - our database is located on the same server with the monitoring system, therefore we connect using localhost;
  4. Database Name: icingaweb2 - is a database name that has been created above;
  5. Username: icingaweb2 - is a username;
  6. Password - in the example above, we used icingapasswd;
  7. Check the validity of the configuration, if successful, continue to the next step.

Configuring Icinga Web 2

Then, there is a need to specify the backend name for our web interface. In other words, it is a name for the data source that has been configured before.

Configuring Icinga Web 2

Specify a username (1) and a password (2) to enter a web interface.

Configuring Icinga Web 2

Then, we are asked to specify logging parameters. Leave the values as they are, on default.

Configuring Icinga Web 2

Then, the system informs about a successful configuration of Icinga Web 2. Continue further.

Configuring Icinga Web 2

We need to configure the monitoring module that is the only module that was enabled by us when configuring Icinga Web 2.

Configuring Icinga Web 2

To do this, configure parameters for connecting to the IDO database (it was created and configured when installing and configuring IDO):

  1. Resource Name: icinga_ido - is the name of the data resource;
  2. Database Type: MySQL - in this article, we use the MySQL base;
  3. Host: localhost - our database is located on the same server with the monitoring system, therefore, we connect using localhost;
  4. Database Name: icinga - is a name of the database that has been created before;
  5. Username: icinga - is a username;
  6. Password - in the example above, we used icinga;
  7. Check the configuration for validity, if successful, continue to the next step.

Configuring Icinga Web 2

Then, we should configure the control method for Icinga via Icinga Web 2:

  1. Transport Name: Icinga2 - is a name;
  2. Transport Type: Icinga 2 API - in the context of this article, we will use API;
  3. Host: loocalhost - both Icinga and Icinga Web 2 are on the same server, therefore we connect to them using localhost;
  4. Port: 5665 - is a default port of Icinga API;
  5. API Username: root;
  6. API Password: a username and a password for connecting to API are specified in the file /etc/icinga2/conf.d/api-users.conf;
  7. Check the configuration for validity, if successful, continue to the next step.

Configuring Icinga Web 2

Then, we see total information about a module configuration:

Configuring Icinga Web 2

A system informs us about the successful completion of the configuration and offers to see a brief log and go to a web interface.

Configuring Icinga Web 2

Then, we are greeted by the homepage of Icinga Web 2, localhost is a part of the configuration on default:

Configuring Icinga Web 2

At this point, the installation of Icinga + Icinga Web 2 is over. This combination is a minimally sufficient one for convenient configuring and monitoring. During the next step, we will configure representing graphs in a web interface and a map that shows that location of hosts.

Building and Representing Graphs Using InfluxDB + Grafana and Icinga Web 2 module Grafana

To save readings of sensors and other information, we are going to use a time series database InfluxDB. To build graphs, we are going to use Grafana, and to represent graphs in а a web interface, we are going to use Icinga Web Grafana module

Download and install InfluxDB:

 

wget https://dl.influxdata.com/influxdb/releases/influxdb_1.7.2_amd64.deb
sudo dpkg -i influxdb_1.7.2_amd64.deb

Then, we need to allow database control via HTTP API (for working with Grafana). To do this, uncomment the next lines in the configuration file InfluxDB - /etc/influxdb/influxdb.conf in the section [http]:

 

enabled = true
bind-address = ":8086"

Install Grafana:

 

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.1.4_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_5.1.4_amd64.deb

And start it:

 

systemctl daemon-reload
systemctl start grafana-server

A web interface of Grafana will be available on the address http://192.168.0.45:3000, data for a default login are admin/admin. In the interface, we need to configure a connection to InfluxDB for building graphs on the basis of the data that are stored in it. To do this, go to Configuration (1) → Data Sources (2) → Add data source (3).

Grafana adding Data Source

Then, choose the data source type:

Grafana adding Data Source

And configure connection parameters as follows:

  1. Name: icinga2 - is a data source name;
  2. URL: http://localhost:8086 - is the address for connecting to the data source. As soon as we have the InfluxDB installed on the same server, we connect to it using localhost;
  3. Database: icinga2influx - is a database name, where readings are stored, which was configured before, the same as a username/password;
  4. User/Password - is a username and a password for connecting to the base. 

Grafana adding Data Source

Afterward, there is a need to create a dashboard, where we need to configure all the necessary graphs for us. We prepared the dashboard for you on the basis of dashboard base-metrics that was taken from the Internet. To import it, there is a need to download base-metrics.json and upload it to Create (1) → Import (2) → Upload .json File (3):

Grafana importing Dashboard

See the configurations of the imported dashboard - leave here everything on default.

Grafana importing Dashboard

After importing, we see the dashboard with the next contents:

Grafana Dashboard

Do not get scared from errors and no data. Such a situation is caused by UniPing server solution v4/SMS not yet connected to the monitoring system, which means that the system does not collect the data received from it.

Then, for Icinga to record the data being obtained to the InfluxDB database, there is a need to enable the option InfluxDB Writer. To do this, write the next command in the console:

 

icinga2 feature enable influxdb

And make /etc/icinga2/features-enabled/influxdb.conf look like:

 

object InfluxdbWriter "influxdb" {
host = "127.0.0.1"
port = 8086
database = "icinga2influx"
username = "icinga2influx"
password = "icinga"
flush_threshold = 1024
flush_interval = 10s
host_template = {
    measurement = "$host.check_command$"
    tags = {
        hostname = "$host.name$"
    }
}
service_template = {
    measurement = "$service.check_command$"
    tags = {
        hostname = "$host.name$"
        service = "$service.name$"
    }
}
enable_send_thresholds = true
enable_send_metadata = true
}

During the next step, there is a need to install Icinga Web 2 module Grafanа to represent graphs from Grafana right to the Icinga web interface. More detailed information about this module can be obtained from the documentation. Let's point out basic moments of the installation. Download:

 

MODULE_VERSION="1.1.7"
ICINGAWEB_MODULEPATH="/usr/share/icingaweb2/modules"
REPO_URL="https://github.com/Mikesch-mp/icingaweb2-module-grafana"
TARGET_DIR="${ICINGAWEB_MODULEPATH}/grafana"
URL="${REPO_URL}/archive/v${MODULE_VERSION}.tar.gz"
install -d -m 0755 "${TARGET_DIR}"
wget -q -O - "$URL" | tar xfz - -C "${TARGET_DIR}" --strip-components 1

And connect it:

 

icingacli module enable grafana

Then, there is a need to allow anonymous read-only access to Grafana API. To do this, make the section[auth.anonymous] in the configuration file /etc/grafana/grafana.ini look like the next:

 

[auth.anonymous]
# enable anonymous access
enabled = true

# specify organization name that should be used for unauthenticated users
org_name = Koebbes

# specify role for unauthenticated users
org_role = Viewer

A further configuration can be continued comfortably in a web interface. Go to the Configuration → Modules → Grafana → Configuration:

  1. Host: localhost:3000 - is a port and address for connection to Grafana;
  2. Protocol: http - protocol;
  3. Default dashboard: base-metrics - dashboard on default, the module will take graphs from it;
  4. Default panel id: 1 - a default graph id;
  5. Default organization id: 1 - id of the "organization" of the anonymous access;
  6. Datasource Backend: Database - data source for graphs;
  7. Datasource Type: InfluxDB - data source type;
  8. Grafana access: iFrame - is a way of representing graphs in the Icinga interface. This type is going to create Iframe and open the part of Grafana interface with a graph in it. This way is the simplest one when installing Icinga and Grafana on the same server and having direct access to it. More detailed information about other ways can be read in the documentation of the module;
  9. Show debug - in case graphs are not appearing in the interface, then you can enable debug mode by checking this checkbox. The debug information will appear next to the supposed location of the graph representation. Using this information, it is possible to conclude the current issue.

Configuring Icinga Web 2 module Grafana

World Map Representing Host Locations

Icinga does not contain the map for representing host locations out of the box, therefore we are going to use the module Icinga Web 2 module map. The installation is simple and described in the documentation

Download it from the git repository:

 

ICINGAWEB_MODULEPATH="/usr/share/icingaweb2/modules"
REPO_URL="https://github.com/nbuchwitz/icingaweb2-module-map"
TARGET_DIR="${ICINGAWEB_MODULEPATH}/map"
git clone "${REPO_URL}" "${TARGET_DIR}"

And enable the module:

 

icingacli module enable map

At this point, the map installation and configuration are over. A web interface gets a new menu:

Icinga world map

Configuring Icinga for Monitoring UniPing server solution v4/SMS and Connected Sensors

The most convenient way is to configure and add hosts to the monitoring system using the console. We are going to be interested in the next three configuration files:

  • /etc/icinga2/conf.d/hosts.conf - host configuration description;
  • /etc/icinga2/conf.d/services.conf - configuration description for monitoring services of different indicators;
  • /etc/icinga2/conf.d/commands.conf - description of monitoring commands

Let's start with /etc/icinga2/conf.d/hosts.conf. Basic moments are commented. Add to the end of the file:

 

object Host "UniPing server solution v4 SMS" { 
import "generic-host"
address = "192.168.0.21"
vars.geolocation = "55.758780, 37.619171" /* Location coordinates of the device. Coordinatres are used by the plugin with a map */
vars.http_vhosts["http"] = {
    http_uri = "/"
}
vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
}

/* Пользовательские переменные */
vars.os = "netping" /* Explicitly indicate that it is a NetPing device, for checking automatic connecting of services and scripts */
vars.snmp_port = "161"
vars.snmp_community = "ping21"

/* configuring monitoring services for temperature, humidity, and IO lines is build in the way that we add service by adding a new variable */

vars.tempsensors["server_room"] = { temp_sensor_n = "1" } /* Monitoring a temperature sensor, vars.tempsensors["sensor_name"] = { temp_sensor_n = "sensor_number" } */
vars.tempsensors["Windows"] = { temp_sensor_n = "2" }

vars.relhumsensors["server_room"] = { relhum_sensor_n = "1" } /* Monitoring a humidity sensor, vars.relhumsensors["sensor_name"] = { relhum_sensor_n = "sensor_number" } */
vars.relhumsensors["Rack 1"] = { relhum_sensor_n = "2" }

vars.iolines["door"] = { io_sensor_n = "1", norm_value = "0", alert_value = "1" } /* Monitoring IO, vars.relhumsensors["sensor_name"] = { io_sensor_n = "IO line number", norm_value = "normal", alert_value = "alert" } */
vars.iolines["air_flow"] = { io_sensor_n = "2", norm_value = "0", alert_value = "1" }
vars.iolines["3"] = { io_sensor_n = "3", norm_value = "0", alert_value = "1" }
vars.iolines["4"] = { io_sensor_n = "4", norm_value = "1", alert_value = "0" }
}

Then, go to /etc/icinga2/conf.d/services.conf. Add to the end of the file:

 

apply Service "npTemp " for (tempsensor => config in host.vars.tempsensors) {
    import "generic-service"

    vars += config

    display_name = "Temperature " + vars.temp_sensor_n + " (" + tempsensor + ")" /* a name that is going to be represented in a web interface */
    check_command = "npTemp" /* a script command that checks it */
    vars.arguments = [ vars.temp_sensor_n ] 
    assign where host.vars.os == "netping"
}

    apply Service "npRelhum " for (relhumsensor => config in host.vars.relhumsensors) {
    import "generic-service"

    vars += config

    display_name = "Relative Humidity " + vars.relhum_sensor_n + " (" + relhumsensor + ")"
    check_command = "npRelhum"
    vars.arguments = [ vars.relhum_sensor_n ]
    assign where host.vars.os == "netping"
}

apply Service "npIo " for (ioline => config in host.vars.iolines) {
    import "generic-service"

    vars += config
    if(!vars.norm_value) {vars.norm_value = "1"}
    if(!vars.alert_value) {vars.alert_value = "0"}

    display_name = "IO " + vars.io_sensor_n + " (" + ioline + ")"
    check_command = "npIo"
    vars.arguments = [ vars.io_sensor_n , vars.norm_value, vars.alert_value ]
    assign where host.vars.os == "netping"
}

And the last one is /etc/icinga2/conf.d/commands.conf. In the configuration, we refer to the scripts that are going to be represented below. Add to the end of the file as follows:

 

object CheckCommand "npTemp" { /* check-command name */
    import "plugin-check-command"
    command = [ PluginContribDir + "/np_temp" ] /* script that is invoked by it */
    arguments = { /* arguments that are passed to the script */
        "-ip" = "$address$"
        "-p" = "$snmp_port$"
        "-c" = "$snmp_community$"
        "-n" = "$temp_sensor_n$"
}
    vars.plugin_address = "$address$"
    vars.snmp_port = "161"
}

object CheckCommand "npRelhum" {
    import "plugin-check-command"
    command = [ PluginContribDir + "/np_relhum" ]
    arguments = {
        "-ip" = "$address$"
        "-p" = "$snmp_port$"
        "-c" = "$snmp_community$"
        "-n" = "$relhum_sensor_n$"
}
    vars.plugin_address = "$address$"
    vars.snmp_port = "161"
}

object CheckCommand "npIo" {
    import "plugin-check-command"
    command = [ PluginContribDir + "/np_io" ]
    arguments = {
        "-ip" = "$address$"
        "-p" = "$snmp_port$"
        "-c" = "$snmp_community$"
        "-n" = "$io_sensor_n$"
        "-normVal" = "$norm_value$"
        "-alertVal" = "$alert_value$"
}
    vars.plugin_address = "$address$"
    vars.snmp_port = "161"
}

Then, download the archive with scripts and unzip the contents to the next directory /usr/lib/nagios/plugins (icinga uses the basic set of plugins for monitoring for Nagios monitoring, and self-written plugins from Nagios are compatible with Icinga).

Then, restart Icinga:

 

systemctl restart icinga2

In a web interface in Overview (1) → Hosts (2) → UniPing server solution v4/SMS (3) → Services (4), we see the list of all our added services and default services:

Icinga UniPing server solution v4SMS services

As soon as you see, the system issues alert using three sensors on IO lines, about the absence of temperature sensors and humidity sensors on the №2 (in fact, there are no sensors there, they were added so that it is possible to visualize adding process and exceeding a relative humidity. In addition, there are two default services on the list - http and ping (They are also used to determine host alive). then, go to configuring graph representation. At this step, a web interface Grafana should show the values on graphs to you because the scripts that gather data have started working and recording the information to the database.

Grafana Dashboard

To match a graph from Grafana and a service from Icinga, we need to know a full name of the service, a dashboard name (base-metrics) and the id of the graph that is required. Let's examine the issue using the example of a humidity sensor built in the temperature sensor. We can take a graph Id from the url address of the webpage with it. A temperature graph of the humidity sensor has id = 17:

Grafana graph id

Then, go to the Web interface Icinga → Configuration → Grafana Graphs → Add New Grafana Graph and:

  1. Name: npRelhum server_room - is a graph name, must coincide with a service name;
  2. Dashboard name: base-metrics - is a dashboard name in Grafana, where the graph is located;
  3. PanelId(s): in this case, 17, 13 - are graph ids: 17 - is an id of the temperature of a humidity sensor, and 13 is humidity.

Icinga Web 2 module Grafana configuring a graph

Now, if we go to Overview (1) → Hosts (2) → UniPing server solution v4/SMS → Relative Humidity 1 (server_room), then we will see graphs with readings on the right of the interface.

Icinga viewing graphs

In this way, there is a need to add all graphs for all services. A separate item is WorldMap with all its devices. A variable for the host is vars.geolocation = "55.758780, 37.619171" as soon as we indicated its location, and the map has the next view:

Icinga world map

Conclusion

The Icinga monitoring system is a perfect fit for daily tasks of monitoring the IT infrastructure of both big and small organizations. The system is flexible and expandable: you can write a plugin for monitoring of nearly everything (of course, if this plugin is not already written). A basic set is deployed in a few console commands, and auxiliary plugins and modules really expand the functionality of the system and allow integrating it with such products as InfluxDB, Grafana, and others. Regardless of the necessity to configure a bigger part of parameters using the console, Icinga does not seem to be difficult. Overall the Icinga monitoring system is a full-fledged and convenient product and it can be eagerly recommended for using.


Tags: All devices
comments powered by Disqus