checkmk increase the max_check_attempts from 1 to 3

by default checkmk send alert if the host failed with 1 check attempt. It's is too senscenity for most of the environment. Here is how to increase the max_check_attempts from 1 to 3.

How to open check_mk port to specific IP address on CentOS 8

I installed the latest check_mk agent on a CentOS 8 server.

dnf install check-mk-agent-2.0.0p13-1.noarch.rpm

To open check_mk port to specific IP address ( for example: 10.10.10.10 is my check_mk host server ) on CentOS 8 server.

firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="10.10.10.10/32"
  port protocol="tcp" port="6556" accept'

Always reload firewalld after changes

firewall-cmd --reload

Increase checkmk check interval to a longer interval in order to save CPU resources on the server and target systems

Try to lower the frequency of checkmk runs.

• Search check interval from the "Setup" from checkmk interface

• Create a rule for "Normal check interval for service checks"

• Create a rule for "Normal check interval for host checks

References:

https://docs.checkmk.com/latest/en/monitoring_basics.html
https://forum.checkmk.com/t/solved-change-check-interval-of-my-own-datasource-program-checks/20312
https://forum.checkmk.com/t/check-mk-english-how-to-set-frequency-of-mk-apt-plugin/12123

Amzaon Linux syslog error: CRIT amazon-ssm-agent: caused by: EC2MetadataError: failed to make EC2Metadata request

Recently got this error from syslog: CRIT amazon-ssm-agent: caused by: EC2MetadataError: failed to make EC2Metadata request It is a EC2 Instance not management by amazon-ssm.

AWS Systems Manager Agent (SSM Agent) is Amazon software that can be installed and configured on an Amazon EC2 instance, an on-premises server, or a virtual machine (VM). SSM Agent makes it possible for Systems Manager to update, manage, and configure these resources. The agent processes requests from the Systems Manager service in the AWS Cloud, and then runs them as specified in the request.

It is safe to drop the package amazon-ssm-agent.

First check and stop the status:

# systemctl status amazon-ssm-agent.service

● amazon-ssm-agent.service - amazon-ssm-agent
   Loaded: loaded (/usr/lib/systemd/system/amazon-ssm-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-10-28 17:04:27 EDT; 1 day 3h ago
 Main PID: 3285 (amazon-ssm-agen)
   CGroup: /system.slice/amazon-ssm-agent.service
           └─3285 /usr/bin/amazon-ssm-agent

Oct 28 23:39:27 local amazon-ssm-agent[3285]: ...

# systemctl stop amazon-ssm-agent.service

Drop the package:

# yum erase amazon-ssm-agent

Loaded plugins: etckeeper, extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package amazon-ssm-agent.x86_64 0:2.3.662.0-1.amzn2 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
 Package                              Arch                       Version                                  Repository                       Size
================================================================================================================================================
Removing:
 amazon-ssm-agent                     x86_64                     2.3.662.0-1.amzn2                        @amzn2-core                      61 M

Transaction Summary
================================================================================================================================================
Remove  1 Package

Installed size: 61 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
etckeeper: pre transaction commit
  Erasing    : amazon-ssm-agent-2.3.662.0-1.amzn2.x86_64                                                                                    1/1
etckeeper: post transaction commit
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 597 bytes | 597.00 KiB/s, done.
Total 6 (delta 5), reused 0 (delta 0)
  Verifying  : amazon-ssm-agent-2.3.662.0-1.amzn2.x86_64                                                                                    1/1

Removed:
  amazon-ssm-agent.x86_64 0:2.3.662.0-1.amzn2

Complete!

How to reset check_mk admin password

My Windows 10 Laptop just crashed after latest Intel patch upgrade.

Forgot to save the admin password on one of the latest check_mk site I built.

su - mysite

cd etc

ls -l htpasswd

OMD[mysite]:~/etc$ ls -l htpasswd

Example: check_mk version 1.4:

lambert:!$1$042926$lmAEb.P5TAi0sZfiGyWHX.
omdadmin:M29dfyFjgy5iA

check_mk version 1.5:

automation:$1$857531$PzhNz/zMdQv8xGu68EZhD.
cmkadmin:$1$386620$Xe5mZKwrPRlzx0BvZCJM8.

htpasswd htpasswd omdadmin          # set password for user *omdadmin*

New password: 

Re-type new password:

You should be good! Have a nice day!

How to Upgrade check-mk to latest version on Debian 8 jessie

You don't need shutdown before the dpkg package upgrade.

1. Download and upgrade to the latest check-mk package:

# dpkg -l|grep check-mk
ii  check-mk-agent                 1.2.8p15-1                         all          Check_MK Agent for Linux
ii  check-mk-raw-1.4.0p8           0.jessie                           amd64        Check_MK is a full featured system monitoring

# wget https://mathias-kettner.de/support/1.4.0p36/check-mk-raw-1.4.0p36_0.jessie_amd64.deb

# sudo dpkg -i check-mk-raw-1.4.0p36_0.jessie_amd64.deb 
Selecting previously unselected package check-mk-raw-1.4.0p36.
(Reading database ... 54001 files and directories currently installed.)
Preparing to unpack check-mk-raw-1.4.0p36_0.jessie_amd64.deb ...
Unpacking check-mk-raw-1.4.0p36 (0.jessie) ...
Setting up check-mk-raw-1.4.0p36 (0.jessie) ...
New default version is 1.4.0p36.cre.
update-alternatives: using /omd/versions/1.4.0p36.cre to provide /omd/versions/default (omd) in manual mode
Installing systemd service: omd.init
Disabling startup through SysV (/etc/init.d/*) scripts
Activating startup during system boot
Processing triggers for systemd (215-17+deb8u7) ...
Processing triggers for man-db (2.7.0.2-5) ...

2. After package upgrade su to omd site user to perform site update:



# su - milllion-doller-server
OMD[milllion-doller-server]:~$ omd status
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
crontab:        running
-----------------------
Overall state:  running

OMD[milllion-doller-server]:~$ omd update
Please completely stop 'milllion-doller-server' before updating it.

OMD[milllion-doller-server]:~$ omd stop
Removing Crontab...OK
Stopping dedicated Apache for site milllion-doller-server....OK
Stopping nagios......OK
Stopping npcd...OK
Stopping rrdcached...waiting for termination...OK
Stopping mkeventd...killing 21922....OK
OMD[milllion-doller-server]:~$ omd update
Unmounting temporary filesystem...OK
Cleaning up temp filesystem...OK

+-------------------------------------------------------------+
| You are going to update the site milllion-doller-server     |
| from version 1.4.0p8.cre to version                         |
| 1.4.0p36.cre. This will include updating all of             |
| you configuration files and merging changes in              |
| the default files with changes made by you. In              |
| case of conflicts your help will be needed.                 |
+-------------------------------------------------------------+
|                 < Abort >                          |
+-------------------------------------------------------------+


2018-09-28 13:32:09 - Updating site 'milllion-doller-server' from version 1.4.0p8.cre to 1.4.0p36.cre...

 * Updated        etc/nagvis/nagvis.ini.php
 * Updated        etc/nagvis/conf.d/demo.ini.php
Finished update.

3. Start the omd site after upgrade and check the status:



OMD[milllion-doller-server]:~$ omd start
Creating temporary filesystem /omd/sites/milllion-doller-server/tmp...OK
Starting mkeventd...OK
Starting rrdcached...OK
Starting npcd...OK
Starting nagios...2018-09-28 13:32:15 [6] updating log file index
2018-09-28 13:32:15 [6] updating log file index
OK
Starting dedicated Apache for site milllion-doller-server...OK
Initializing Crontab...OK

OMD[milllion-doller-server]:~$ omd status
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
crontab:        running
-----------------------
Overall state:  running

check_mk agent install on CentOS 7 by systemd

Here is the steps I install check_mk agent on CentOS 7 by systemd only without xinit.

first scp the rpm package to CentOS server, then install it:
rpm -ivh check-mk-agent-1.2.8p18-1.noarch.rpm

Open the port 6556:

firewall-cmd --permanent --add-port=6556/tcp
firewall-cmd --reload

Double check the status:

firewall-cmd --list-ports
firewall-cmd --list-services

Restart to make sure the check_mk_agent start on boot.

Hope this helps.

check_mk hang on start with mkeventd

1. Starting mkeventd (builtin: snmptrap)...Cannot bind UDP socket for snmptrap to port: Permission denied

chmod u+s /opt/omd/sites/mysite/bin/mkeventd_open514
omd restart

2. It just hang and waiting forever on Starting mkeventd (builtin: snmptrap)...

I found a lot files in /opt/omd/sites/(sitename)/var/mkeventd/status.

Have to manually delete the status and history files to start the mkeventd.

check_mk How to fix:Disk Mount Changes showing error

Sometimes few hosts in my check_mk build showing Disk Mount error.

For example:
"Mount options of / - - - - CRIT - mount options are
> > > acl,barrier=0,data=ordered,errors=remount-ro,noatime,rw,user_xattr,
> > > expected are barrier=0,data=ordered,errors=remount-ro,noatime,rw"

But there isn't any changes recently.

I just su to my site and refresh the hosts manually:

# su - [site_name]
# cmk -II [$host_name]
# cmk -R

Restart the check_mk, that will make the nonsense error goes away.

check_mk ALL Hoststate down, check_icmp: Failed to obtain ICMP socket:Operation not permitted

I upgrade my production check_mk site to the latest 1.2.8p14.

Suddenly all Hosts shows down.

/opt# find . -name check_icmp
./omd/versions/1.2.8p13.cre/lib/nagios/plugins/check_icmp
./omd/versions/1.2.8p13.cre/share/check_mk/checks/check_icmp
./omd/versions/1.2.8p13.cre/share/check_mk/checkman/check_icmp
./omd/versions/1.2.8p14.cre/lib/nagios/plugins/check_icmp
./omd/versions/1.2.8p14.cre/share/check_mk/checks/check_icmp
./omd/versions/1.2.8p14.cre/share/check_mk/checkman/check_icmp

root@CHECK-MK:/opt# chmod u+s ./omd/versions/1.2.8p14.cre/lib/nagios/plugins/check_icmp
root@CHECK-MK:/opt# chmod u+s ./omd/versions/1.2.8p14.cre/share/check_mk/checks/check_icmp
root@CHECK-MK:/opt# chmod u+s ./omd/versions/1.2.8p14.cre/share/check_mk/checkman/check_icmp

Restart the site, everything back to normal.

This happens after I update check_mk to the latest stable Version: 1.2.8p14
Hope it helps.

check_mk outdate error message: Disk Mount Changes network interfacespeed wrong

A lot time I got these kind of wrong error message like:

Disk Mount changes but missing mount option.
Or Network interface 1 is 10GB but should be 1 GB.

Which is just the check_mk output outdate issue.

Here is what I do to refresh the $host in check_mk:

# su - monitorsite_name
# cmk -II $host
# cmk -R

Sometimes I have to restart omd site, but it should be enough to give you back the correct output.

check_mk agent start by systemd Debian Jessie

I use check_mk everywhere.

For the old school, on Debian system you need install xinetd to start it.
But since we have systemd in Debian 8 Jessie, it will be much cleaner to do systemd way.

Here is how I do it:

1. Fist if you previous using xinetd, stop and remove it.



# service xinetd stop
# apt-get purge xinetd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfile-copy-recursive-perl libx265-79 update-inetd

Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  xinetd*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 255 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 34520 files and directories currently installed.)
Removing xinetd (1:2.3.15-3) ...
Purging configuration files for xinetd (1:2.3.15-3) ...
Processing triggers for man-db (2.7.0.2-5) ...


And run autoremove to cleanup


# apt-get autoremove

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libfile-copy-recursive-perl libx265-79 update-inetd
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 3,244 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 34490 files and directories currently installed.)
Removing update-inetd (4.43) ...
Removing libfile-copy-recursive-perl (0.38-1) ...
Removing libx265-79:i386 (1.9-3~bpo8+1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for libc-bin (2.19-18+deb8u6) ...



2. Then check your check_mk agent version:

# dpkg -l|grep check-mk
ii  check-mk-agent                      1.2.8p12-1                        all          Check_MK Agent for Linux



3. Next enable the check_mk in systemd autostart.



# systemctl enable check_mk.socket
Created symlink from /etc/systemd/system/sockets.target.wants/check_mk.socket to /etc/systemd/system/check_mk.socket.



4. Finally start the service.



# systemctl start check_mk.socket


A special notice, because the systemd not allow the IP rules like xinetd does.
You will have to update your firewall to restrict other accessing.

Hope this helps, please let me know if you have any questions.

Increase SNMP ping Timeout Period for Check_MK

If you are monitoring external devices through SNMP like Netapp Storage, it may throw out alert when the Device is busy or the connect time out.

Put this in your main.mk file:

su - [OMDSiteName Example:] mds
vi ~/etc/check_mk/main.mk

snmp_timing = [
 ( {'retries': 2, 'timeout':30}, ["snmp"], ALL_HOSTS ),
]

Let check_mk server check 1 more time before it send alert after the first one fail.
The wait time out increase to 30 seconds.
These lines apply changes on all of your SNMP machines having the tag snmp.

To force checm_mk client and ping run second checking if first one fail, add these into ~/etc/check_mk/main.mk:

extra_service_conf['max_check_attempts'] = [
  ("2", ALL_HOSTS, "Check_MK"),
  ("3", ALL_HOSTS, "PING"),
]

Then restart OMD/Check_MK to make it works.

OMD[mds]:omd restart

To make all check_mk less sensitive:

extra_service_conf['max_check_attempts'] = [
  ("2", ALL_HOSTS, ALL_SERVICES),
  ("3", ALL_HOSTS, "PING"),
]

Reference: http://mathias-kettner.de/checkmk_configvars.html
http://lists.mathias-kettner.de/pipermail/checkmk-en/2010-December/002241.html

OMD ssmtp - extremely simple MTA to get mail off the system to a mailhub

ssmtp - extremely simple MTA to get mail off the system to a mail hub

apt-get purge exim4*
apt-get install ssmtp bsd-mailx
mail -s "OMD test email 7" -S from=test@milliondollarserver.com test@milliondollarserver.com

mail -s "OMD test email 7" test@milliondollarserver.com

root@monitor:/etc/ssmtp# cat ssmtp.conf

#

# Config file for sSMTP sendmail

#

# The person who gets all mail for userids < 1000

# Make this empty to disable rewriting.

root=postmaster



# The place where the mail goes. The actual machine name is required no

# MX records are consulted. Commonly mailhosts are named mail.domain.com

UseTLS=YES

UseSTARTTLS=Yes

mailhub=smtp.gmail.com:587

AuthUser=test@milliondollarserver.com

AuthPass=IFUseeTHISpassWord

AuthMethod=LOGIN



# Where will the mail seem to come from?

#rewriteDomain=



# The full hostname

hostname=monitor.milliondollarserver.com



# Are users allowed to set their own From: address?

# YES - Allow the user to specify their own From: address

# NO - Use the system generated From: address

FromLineOverride=YES

Reference:
https://wiki.debian.org/sSMTP
http://www.debiantutorials.com/installing-ssmtp-mta-mail-transfer-agent/

Monitoring Windows Service with Check_MK

I have to Monitoring Windows Service with Check_MK.

The in-house windows application run as a service on Windows Server.

Sometime it crashed without any notification, just need monitoring it by Check_MK.

1. Check the service name by cmk command:


Run it from the OMD check_mk box:
For my site:(mdsmonitor):




/omd/sites/mdsmonitor/bin/cmk -d [Windows Host need to be monitor] |grep [Crasher_service]




The output should be something like:




Crasher_service running/auto mds_in_house_service





2. Add service name into main.mk file:


add 1 line into /opt/omd/sites/mdsmonitor/etc/check_mk/main.mk



inventory_services = ['Crasher_service']




3. Restart OMD to adapt the changes:





su - mdsmonitor
omd restart




4. Re Scan the service on Host, add the new service from "Missing Services"

Reference:https://mathias-kettner.de/checkmk_windows.html

Centos 7 check_mk agent quick install

1. First Install the package:




yum install check-mk-agent



2. Then Change the /etc/xinetd.d/check-mk-agent:



# configure the IP address(es) of your Nagios server here:
only_from      = 127.0.0.1 192.168.1.31



3. Start xinetd service:



systemctl start  xinetd.service
systemctl enable xinetd.service



4. Allow firewalld on port 6556:



firewall-cmd --add-port=6556/tcp --permanent
firewall-cmd --reload