Monitoring on Janik von Rotz

Monitor cron jobs with Prometheus, Grafana and Node exporter

Mon, 07 Sep 2020 11:26:05 +0200

Nobody wants to be notified by email anymore, especially if its a failed cron job. We have advanced monitoring systems that tell if somethings wrong. In my case I use Grafana and Prometheus and Node exporter to collect host metric, visualize them and send out alerts. Usually, one would set up an exporter to monitor an new piece of software, but for cron there isn’t any exporter available. In contraire there are a lot of online service to monitor your cron jobs, such as Cronitor.io. But we do not want to add another dependency for simply monitoring cron jobs.

In this tutorial I will elaborate on how I look after cron jobs with Prometheus and Grafana. We are going to configure the textfile collector of the Node exporter, define custom metrics and visualize them in a Grafana dashboard.

I assume that there is machine running with cron jobs. This machine has multiple cron jobs and a configured Node exporter. The Node metrics are scrapped by Prometheus and visualized in Grafana.

First, we are going to add a bash script to write custom Node exporter metrics. Copy the script below to the host.

/usr/local/bin/write-node-exporter-metric

#!/bin/bash

# Display Help
Help() {
    echo
    echo "write-node-exporter-metric"
    echo "##########################"
    echo
    echo "Description: Write node-exporter metric."
    echo "Syntax: write-node-exporter-metric [-n|-c|-v|help]"
    echo "Example: write-node-exporter-metric -n cron_job -c \"Renew certs for proxy01\" -v 0"
    echo "options:"
    echo "  -n    Reference of custom metric type. Defaults to 'cron_job'"
    echo "  -c    Code for metric value."
    echo "  -v    Value of metric."
    echo "  help  Show write-node-exporter-metric help."
    echo
}

# Show help and exit
if [[ $1 == 'help' ]]; then
    Help
    exit
fi

# Process params
while getopts ":n :c: :v:" opt; do
  case $opt in
    n) TYPE="$OPTARG"
    ;;
    c) CODE="$OPTARG"
    ;;
    v) VALUE="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    Help
    exit;;
  esac
done

# Fallback to environment vars and default values
: ${TYPE:='cron_job'}

[[ -z "$CODE" ]] && { echo "Parameter -c|code is empty" ; exit 1; }
[[ -z "$VALUE" ]] && { echo "Parameter -v|value is empty" ; exit 1; }

if [ "$TYPE" == "cron_job" ]; then
    echo "Write metric node_cron_job_exit_code for code \"$CODE\" with value $VALUE."
    ID=$(echo $CODE | shasum | cut -c1-5)

    cat << EOF >> /var/tmp/node_cron_job_exit_code.$ID.prom.$$
# HELP node_cron_job_exit_code Last exit code of cron job.
# TYPE node_cron_job_exit_code counter
node_cron_job_exit_code{code="$CODE"} $VALUE
EOF
    mv /var/tmp/node_cron_job_exit_code.$ID.prom.$$ /var/tmp/node_cron_job_exit_code.$ID.prom
fi

And make it executable.

chmod +x /usr/local/bin/write-node-exporter-metric

By default this script writes metric text files to /var/tmp. This folder is watched by Node exporter. Set the textfile collector directory flag --collector.textfile.directory for the Node exporter. If you are using Docker to run the exporter, set the following config:

...
volumes:
  - /:/hostfs
command: '--collector.textfile.directory=/hostfs/var/tmp'
...

Let’s write a custom metric and see if it scrapped by Prometheus.

Run write-node-exporter-metric -c 'Renew certs for proxy01' -v 0 on the command line.

Check the metrics interface of the host and search for node_cron_job_exit_code.

Use this curl command if you want to stick to the console:

curl --silent --user username:password \
  https://host.example.com/node-exporter/metrics | \
  grep node_cron_job_exit_code

If the value has been exposed, open Grafana and explore the metrics.

Create a new panel and use this query:

sum by (instance) (node_cron_job_exit_code)

This query sums all cron jobs exit codes by instance. If the sum is not null something went wrong.

Create an alert that triggers if the metric is greater than 0.

When setting up cron jobs crontab -e from now on you simply have to add the write metric command at end of the line. Here is an example:

45 0 * * 0 /usr/share/cerbot/renew-certs; write-node-exporter-metric -c 'Renew certs for proxy' -v $?

No matter if the job succeeds or fails, the exit code is written and forwarded to Prometheus.

What do you think? Do you like this solution? Let me know how you monitor cron jobs.

System Observability and Chaos Engineering

Sun, 16 Sep 2018 14:58:02 +0200

At the last Jazoon tech days I learned about the current state of DevOps and other related topics. During the talks “chaos engineering” and “observability” were mentioned many times. I didn’t know either and became curious what it was about.

Observability

TL:DR; Monitoring tells you whether a system is working, observability lets you ask why it isn’t working.

A very insightful presentation on observability was given by Charity Majors.

In her talk she covered the shortcomings of traditional metrics and logs. She pointed out that monitoring as we know it is dead (including dashboards), and that we need to focus on observability.

Enhancing system observability for a software project requires a culture shift rather than implementing a new technology. Software projects become more complex and thus we cannot detect and prevent any kind of error that occurs in the future. Increased observability helps finding, debugging and fixing an issue more easily.

On an abstract level the following question provides a good viewpoint on what observability is about.

Can you understand what’s happening inside your system, just by asking questions from the outside?
– Charity Majors

If the system is a blackbox it becomes very difficult to understand its behavior. That might sound obvious, but of course it is impossible to build a complex system without unknown components.

One of the drivers that makes monitoring obsolete and supports the idea of observability is the unpredictability of system failures. The next quotes puts it very well.

Distributed systems haven an infinitely long list of almost-impossible failure scenarios that make staging environments particularly worthless.
– Charity Majors

This fact has probably lead us to the discipline of Chaos Engineering.

Chaos Engineering

TL:DR; Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Aaron Blohowiak worked as a Chaos Engineer at Netflix and showed how their multi-region infrastructure deals with outages.

The fundamental idea of chaos engineering is already part of the company’s culture.

In general, freedom and rapid recovery is better than trying to prevent error. We are in a creative business, not a safety-critical business.
– jobs.netflix.com/culture

During a back stage talks Aaron talked about Chaos Engineering in practice. The goal of Chaos Engineering is the improvement of system resilience. Instead of practicing error prevention and you focus on running experiments with your system and make sure that it is able to recover from failures.

Due to the inevitability of errors, investing in isolation, recovery and remediation is superior to investing in prevention.
– Aaron Blohowiak

Once system resilience improves, the system can withstand unpredictable errors in production.

If you are interested in this topic start reading the Principles of Chaos Engineering manifesto.

Hope you learned as much as I did! 😃💡 Otherwise leave a question in the comments!

Monitor and audit Active Directory user and group management

Thu, 12 Oct 2017 15:54:08 +0000

Traceability is key when collaborating in the Active Directory (AD). Multiple admins changing and updating permissions and policies makes it difficult being compliant with the company’s policies. It is important to monitor mutations in the directory. By default audit policies are disabled for Domain Controllers (DC) and must be enabled explicitly. Enabling auditing for the DCs is quite easy, querying the logs for a specific event is a bit more difficult.

In this guide you’ll learn how to enable auditing for a specific case and how to query the audit logs for a specific event.

The tutorial assumes that there is a:

Domain Controller.
Group policies, security groups, users, …
Admins with DC access.

Enable Auditing

Let’s start by have a look on the already enabled audit categories.

Log into the DC.
Open PowerShell as admin.
Run auditpol /get /category:*

The command returns a list of audit categories and its status. These settings have been enabled by either the auditpol tool or via GPOs.

In our scenario we would like to track management of users and groups, which is part of the Audit Account Management. To enable this audit category create a new group policiy for the DC.

Open the GPO management console.
Right-click the Domain Controllers organizational unit.
Create new GPO and open it in the GPO editor.
Enable logging for subcategories: Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Security Options > Audit: Force audit policy subcategory settings...
Increase the security log size to 4GB: Computer Configuration > Windows Settings > Security Settings > Event Log > Maximum security log size: 4268032
Then navigate to Computer Configuration > Windows Settings > Security Settings > Advanced Audit Policy Configuration > Account Management
Enable the required audit categories.
Make a gpupdate /force on the DC.
Run auditpol /get /Category:* and double-check whether the settings are correct.

If you open the security event log on the DC there should be events logging account management mutations.

Source: Microsoft Docs - Monitoring Active Directory for Signs of Compromise

Query Audit Logs

As mentioned querying the event log is a bit more difficult. The event log viewer offers limited features for filtering events and searching by specific keywords. In contrast with PowerShell it is possible to filter and search the event log by any property and keyword.

Here is a simple example:

$LogName = "security"
$StartTime = Get-Date("2017-10-12 12:50")
$EndTime = Get-Date("2017-10-12 13:00")
$SearchKey = "username"

Get-WinEvent -FilterHashtable @{LogName=$LogName; StartTime=$StartTime;EndTime=$EndTime} | Where-Object {$_.Message -match $SearchKey} | select Id, TimeCreated, Message | Format-List

Source: Hey, Scripting Guy! Blog - Filtering Event Log Events with PowerShell

Monitoring a SharePoint Environment with Zabbix

Mon, 14 Apr 2014 15:16:08 +0000

This post of is part of my Install SharePoint 2013 Three-tier Farm project.

Zabbix is an OpenSource monitoring tool. And offers everything from simple storage availabilty reporting up to end to end web service calls. It’s extensible with plugins and shell scripts. There isn’t a thing you can’t monitor with Zabbix.

In our company we use Zabbix to monitor the SharePoint infrastructure. One of the great features for this case is the web monitor. Zabbix can request specific websites and authenticate against the webserver. That’s why I don’t need an IIS warm up script for our SharePoint infrastructure. Zabbix always makes a request against the SharePoint website after an IIS recycle.

In case you already have a running Zabbix server you can monitor your SharePoint web application with this web rule: