Monitoring your IT infrastructure can be a challenging task today. Particularly because it usually involves multiple hosts and services, distributed on different locations, some of which in the internal network and some available on the Internet.
In order to have clear understanding and awareness of the current state of the IT infrastructure, the following systems must be in place, so that all problems and bottlenecks can be quickly diagnosed and dealt with. Preferably before they cause any real problem in the network and with the services which are being used by your employees or customers.
Monitoring usually involves several sub-systems:
- Availability – Pro-active monitoring on whether all hosts and services are reachable and available to their end users.
- Performance – Tracking down performance counters for the services, applications and hosts, for historical purposes and identifying trends.
- Logs – Passive monitoring on what the applications and hosts are outputting during their normal operations, so that we can detect warnings or problems which cannot be detected by regular active checks or for debugging and consolidating purposes.
- Security – Monitor potential vulnerabilities and activities which might compromise either normal operations or valuable documents and assets within the company.
- Backup – Yes, monitoring backups should be a separated issues, since doesn’t really fit-in any of the previous sub-systems. Monitoring if for the important systems there are backups which are restorable, available and secure – is a separate topic.
Security monitoring, involves systems such as SIEM, Intrusion & extrusion detection and behavioral monitoring and will not be in the scope of this post. We will write about building your own open source SIEM separately.
What we will present here is how to quickly setup your own virtual server, which will be responsible for monitoring availability and performance graphing. This is based on the Icinga2 monitoring system, together with some other plugins, one of which pnp4nagios, the plugin for historical graphing and reporting. We have also created plugins which will provide partial log and backup monitoring. For a more robust log-management solution, such as Kibana, we will write in a separate blog post.
The monitoring system should have few features, deployed out of the box, such as:
- Notifications and alerting.
- Web interface for preview, management and reporting.
- Built-in plugins for agent and agentless monitoring.
In order to install the Monitoring server, what you would need to have a fresh install of any Debian or Ubuntu variant. We will use salt-stack and a prepared salt-state, in order to install all required packages, configuration files and initial settings.
1. Download automation script. First you need to download our masterless script, which will prepare the server, install salt-minion and other necessary requirements for the salt-state to be applied.
wget https://raw.githubusercontent.com/VapourApps/saltstack/master/masterless.sh
2. Prepare the server. Next we will execute the script to prepare the sever and download all the necessary configuration files, for the monitoring app to be installed and configured.
chmod +x masterless.sh
sudo ./masterless.sh monitoring
3. Set admin password. After the script finishes, the sever is prepared and has downloaded the necessary files to apply the monitoring state. Next we need to modify the configuration file, for initial admin credentials in /srv/pillar/credentials.sls.
vi /srv/pillar/credentials.sls
4. Execute. Then, only run the following command and wait for everything to install and configure
salt-call –local state.highstate
5. Login to web console. After the installation is finished, you can open the IP address of your server in browser and login to the Icingaweb2 monitoring dashboard, with the credentials you have provided.
http://{ip-address-of-virtual-server}, or
http://{ip-address-of-virtual-server}/icingaweb2
6. Add other hosts and services. Now you can configure your hosts and services in /etc/icinga2/conf.d following examples on the Icinga2 documentation website.
7. Set-up notifications. To get realtime notifications, you should change the adminstrator e-mail address in /etc/icinga2/conf.d/users.conf and adjust the outgoing e-mail settings in /etc/ssmtp/ssmtp.conf.
After the changes, only run:
/etc/init.d/icinga2 reload
And… congratulations, you have a fully functional monitoring server, with only a few commands. In a next blog post, we will cover, how to easily add hosts and services on the command line or through our VapourApps Dashboard.
