Grafana on Janik von Rotz

Nginx, Loki, Promtail and Grafana

Tue, 09 Nov 2021 09:13:06 +0100

Hey devops engineer, you don’t need Logtail, Sentry, Datadog or any other SaaS/PaaS service to manage your logs. Collecting and analyzing log files is super easy with the LPG-stack. Another acronym that stands for Loki, Promtail and Grafana.

I will give you a brief overview of how you can deploy the LPG-stack and label your log entries with Promtail. To sum it up we will have look at Grafana and see how we can query the log data

Usually you would follow these four steps to setup the log monitoring system:

Setup loki for indexing log data
Setup nginx reverse proxy to expose loki with basic auth
Configure promtail and forward logs on host
Create a dashboard in grafana and query the data

However, giving details on the full setup would make this post unnecessarily long. That is why I added a note Not covered: note to every section. The deployment of these services depends heavily on the technology (f.g. Kubernetes, Docker Compose or Ansible) you are using. I will fokus on the most important part - the config files.

Loki

Not covered: Deployment and configuration of the Loki container.

Loki manages the log index. It receives the log files from Promtail and acts as a datasource for Grafana.

The configuration below is based on the official template and has not been altered notably.

local-config.yml

# https://github.com/grafana/loki/blob/v2.3.0/cmd/loki/loki-local-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  wal:
    enabled: true
    dir: /tmp/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Nginx reverse proxy

Not covered: Full nginx config with https server definition.

The Nginx proxy terminates https connections and ensures the connections to Loki are authenticated with basic auth.

The configuration below shows the proxy configuration for Loki.

loki.nginx

location /loki/api/v1 {
    auth_basic "loki";
    auth_basic_user_file /etc/nginx/conf.d/proxies/loki.htpasswd;
    proxy_pass http://loki01:3100/loki/api/v1;
    include /etc/letsencrypt/proxy-params.conf;
}

Promtail

Not covered: Deployment of the Promtail container.

Promtail has access to the log folder of the host machine. It extracts all log data and forwards the content to Loki.

Promtail has been configured to use basic auth and extract Docker log files. Before sending the log files it processes and labels the log lines.

config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://loki:ohs24234ch5vahz5ieVei@monitor.example.com/loki/api/v1/push

scrape_configs:
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      instance: server.example.com
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<container_name>(?:[^|]*[^|])).(?P<image_name>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      #tag:
      stream:
      container_name:
      image_name:
  - output:
      source: output

Ensure docker containers are deployed with these log options:

"LogConfig": {
	"Type": "json-file",
	"Config": {
		"max-file": "3",
		"max-size": "10m",
		"tag": "nginx03|nginx"
	}
},

Grafana

Not covered: Deployment of the Grafana container.

Grafana is our monitoring tool. It visualizes the data from Loki. Getting the data requires settings up a data source.

If the connection is private, there is no need for authentication.

In the Grafana data explorer the log data can be queried:

With queries like rate(({job="containerlogs"} |= "error")[1m]) the frequency of errors within a time range will be returned.

Finally, you could setup an alert for a certain threshold on the frequency quer.

Add Prometheus instance variable to Grafana query

Mon, 19 Apr 2021 18:05:23 +0200

Prometheus stores the instance or host name for every metric. If you want to filter your dashboard data based on the instance name you need a Grafana variable.

Open the Grafana, select your dashboard and navigate to Settings > Variables. Create a new variable:

Name: isntance
Typoe: Query
Lable: Instance
Data source: Prometheus
Query: label_values(instance)
Include All option: true

Check if the preview returns the instances.

Next open a panel and insert {instance=~"$instance"} into your metric like this:

Now all the panel data will be filtered by whatever is set for the variable.

Update 2021-09-06

You can also add filters to the query. Use the label_values({job!="blackbox"},instance) query to return labels for instances that are not monitored by a blackbox exporter.

Monitor cron jobs with Prometheus, Grafana and Node exporter

Mon, 07 Sep 2020 11:26:05 +0200

Nobody wants to be notified by email anymore, especially if its a failed cron job. We have advanced monitoring systems that tell if somethings wrong. In my case I use Grafana and Prometheus and Node exporter to collect host metric, visualize them and send out alerts. Usually, one would set up an exporter to monitor an new piece of software, but for cron there isn’t any exporter available. In contraire there are a lot of online service to monitor your cron jobs, such as Cronitor.io. But we do not want to add another dependency for simply monitoring cron jobs.

In this tutorial I will elaborate on how I look after cron jobs with Prometheus and Grafana. We are going to configure the textfile collector of the Node exporter, define custom metrics and visualize them in a Grafana dashboard.

I assume that there is machine running with cron jobs. This machine has multiple cron jobs and a configured Node exporter. The Node metrics are scrapped by Prometheus and visualized in Grafana.

First, we are going to add a bash script to write custom Node exporter metrics. Copy the script below to the host.

/usr/local/bin/write-node-exporter-metric

#!/bin/bash

# Display Help
Help() {
    echo
    echo "write-node-exporter-metric"
    echo "##########################"
    echo
    echo "Description: Write node-exporter metric."
    echo "Syntax: write-node-exporter-metric [-n|-c|-v|help]"
    echo "Example: write-node-exporter-metric -n cron_job -c \"Renew certs for proxy01\" -v 0"
    echo "options:"
    echo "  -n    Reference of custom metric type. Defaults to 'cron_job'"
    echo "  -c    Code for metric value."
    echo "  -v    Value of metric."
    echo "  help  Show write-node-exporter-metric help."
    echo
}

# Show help and exit
if [[ $1 == 'help' ]]; then
    Help
    exit
fi

# Process params
while getopts ":n :c: :v:" opt; do
  case $opt in
    n) TYPE="$OPTARG"
    ;;
    c) CODE="$OPTARG"
    ;;
    v) VALUE="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    Help
    exit;;
  esac
done

# Fallback to environment vars and default values
: ${TYPE:='cron_job'}

[[ -z "$CODE" ]] && { echo "Parameter -c|code is empty" ; exit 1; }
[[ -z "$VALUE" ]] && { echo "Parameter -v|value is empty" ; exit 1; }

if [ "$TYPE" == "cron_job" ]; then
    echo "Write metric node_cron_job_exit_code for code \"$CODE\" with value $VALUE."
    ID=$(echo $CODE | shasum | cut -c1-5)

    cat << EOF >> /var/tmp/node_cron_job_exit_code.$ID.prom.$$
# HELP node_cron_job_exit_code Last exit code of cron job.
# TYPE node_cron_job_exit_code counter
node_cron_job_exit_code{code="$CODE"} $VALUE
EOF
    mv /var/tmp/node_cron_job_exit_code.$ID.prom.$$ /var/tmp/node_cron_job_exit_code.$ID.prom
fi

And make it executable.

chmod +x /usr/local/bin/write-node-exporter-metric

By default this script writes metric text files to /var/tmp. This folder is watched by Node exporter. Set the textfile collector directory flag --collector.textfile.directory for the Node exporter. If you are using Docker to run the exporter, set the following config:

...
volumes:
  - /:/hostfs
command: '--collector.textfile.directory=/hostfs/var/tmp'
...

Let’s write a custom metric and see if it scrapped by Prometheus.

Run write-node-exporter-metric -c 'Renew certs for proxy01' -v 0 on the command line.

Check the metrics interface of the host and search for node_cron_job_exit_code.

Use this curl command if you want to stick to the console:

curl --silent --user username:password \
  https://host.example.com/node-exporter/metrics | \
  grep node_cron_job_exit_code

If the value has been exposed, open Grafana and explore the metrics.

Create a new panel and use this query:

sum by (instance) (node_cron_job_exit_code)

This query sums all cron jobs exit codes by instance. If the sum is not null something went wrong.

Create an alert that triggers if the metric is greater than 0.

When setting up cron jobs crontab -e from now on you simply have to add the write metric command at end of the line. Here is an example:

45 0 * * 0 /usr/share/cerbot/renew-certs; write-node-exporter-metric -c 'Renew certs for proxy' -v $?

No matter if the job succeeds or fails, the exit code is written and forwarded to Prometheus.

What do you think? Do you like this solution? Let me know how you monitor cron jobs.

Grafana OAuth with Keycloak and how to validate a JWT token

Thu, 27 Aug 2020 17:18:30 +0200

In this tutorial I am going to show how you can connect a Garafana container that is hidden behind proxy with Keycloak. We want to log into Grafana with a Keycloak user and experience a seamless SSO-flow. Therefore we are going to configure an OAuth client for Grafana.

For this tutorial I assume that our two services are reachable from a public domain.

Keycloak: login.example.com
Grafana: monitor.example.com

Replace these domains with your case.

Setup Keycloak

First we are going to create a new Keycloak client. I assume Keycloak is already running and a realm has been configured.

Client ID: monitor.example.com
Client Protocol: openid-connect
Root URL: https://monitor.example.com

Further make these configurations:

Access Type: confidentials // The OAuth client must use a client id and secret.
Root URL: ${authBaseUrl}
Valid Redirect URIs: https://monitor.example.com/login/generic_oauth
Base URL: /login/generic_oauth
Clear Admin URL and Web Origins.

Click save and open the Credentials tab. Copy the Secret into a separate note, we will need it in the second and third part of this tutorial.

Open the tab Roles and click Add Role. Create a new role with name admin. This role defines the access level for Grafana.

Assign the client role to your Keycloak user.

Header over to Scope tab and set Full Scope Allowed to OFF. We do not want to share any other details about the realm in the client token.

Finally, we are going to configure a client mapper for the roles property. We must ensure that Grafana can extract the access role from the JWT token. Open the Mappers tab and click on Create. Create an entry with these options:

Name: Roles
Mapper Type: User Client Role
Client ID: monitor.example.com
Token Claim Name: roles
Claim JSON type: string

In the next step we are going to verify that Grafana can retrieve a valid access token.

Verify JWT token

Open your shell, enter the command below and populate the <>-fields. Copy the client secret from the note.

KEYCLOAK_USERNAME=<Keycloak username>
KEYCLOAK_PASSWORD=<Keycloak password>
KEYCLOAK_REALM=<Keycloak realm name>
KEYCLOAK_CLIENT_SECRET=<Keycloak client secret>
curl -s \
-d "client_id=monitor.example.com" \
-d "client_secret=$KEYCLOAK_CLIENT_SECRET" \
-d "username=$KEYCLOAK_USERNAME" \
-d "password=$KEYCLOAK_PASSWORD" \
-d "grant_type=password" \
"https://login.example.com/auth/realms/$KEYCLOAK_REALM/protocol/openid-connect/token" | jq -r '.access_token'

Then copy the encoded output, open https://jwt.io#debugger-io and paste it into the left box. On the rights side you should find the decoded JSON output with this property:

"roles": [
    "admin"
],

This means the client role has been added to the JWT token and mapped correctly .

Grafana’s generic OAuth can be configured to look for this property using a JMESPath. Open this site, paste the decoded output of the JWT token and enter this filter:

contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'

The results box should say Admin.

Grafana

We assume that the Grafana container is running and needs to be configured for OAuth access.

My first choice of configuring any container is using environment variables. Luckily all Grafana settings can be set using environment variables.

Make the following configurations for the Grafana container:

GF_SERVER_DOMAIN: "monitor.example.com"
GF_SERVER_ROOT_URL: "https://monitor.exmpale.com"
GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
GF_AUTH_GENERIC_OAUTH_NAME: "Login Keycloak"
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: "true"
GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "monitor.example.com"
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "$KEYCLOAK_CLIENT_SECRET"
GF_AUTH_GENERIC_OAUTH_SCOPES: profile
GF_AUTH_GENERIC_OAUTH_AUTH_URL: "https://login.example.com/auth/realms/example.com/protocol/openid-connect/auth"
GF_AUTH_GENERIC_OAUTH_TOKEN_URL: "https://login.example.com/auth/realms/example.com/protocol/openid-connect/token"
GF_AUTH_GENERIC_OAUTH_API_URL: "https://login.example.com/auth/realms/example.com/protocol/openid-connect/userinfo"
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

Test

Once everything is deployed logout of Grafana and click on the Login Keycloak button below the login form. You will be forwarded to Keycloak. Keycloak will check the redirect url and client key of the request. If everything looks good to go, you should see the Keycloak login form. Login using the Keycloak user and password and you should be redirected back to Grafana on a successful login. Grafana will create a user if it does not already exist.

Source

Here are the articles that helped me creating this tutorial:

Grafana on Janik von Rotz

Nginx, Loki, Promtail and Grafana

Loki

Nginx reverse proxy

Promtail

Grafana

Add Prometheus instance variable to Grafana query

Monitor cron jobs with Prometheus, Grafana and Node exporter

Grafana OAuth with Keycloak and how to validate a JWT token

Setup Keycloak

Verify JWT token

Grafana

Further readings

Source