Ansible on Janik von Rotz

Ansible: Combine group and host vars

Mon, 25 Oct 2021 08:13:01 +0200

In Ansible the configuration parameters are usually separated into two categories: host and group vars. As their names say these are variables that are either applied to a specific host or a group of hosts. And here is already an edge case. What if we want to configure a variable for group, but then also give the possibility to modify for a specific host. Read on to see my solution.

This use case occurs usually for list vars. In this example we have a list of packages that need to be installed for a host. We have default group setting the packages var, but then there is also a host specific variable.

This is the group var:

inventory/group_vars/all.yml

packages:
  - name: git
  - name: dnsutils
  - name: zsh

For the host var prefix the var with host_. This helps to make a distinction and of course not overwriting the group var.

inventory/host_vars/server1.yml

host_packages:
  - name: cifs-utils

If the variable is list of dictionaries, it is easy to merge the two variables:

roles/packages/tasks/main.yml

- name: Combine group and host vars
  set_fact:
    packages: "{{ packages + host_packages }}"
  when: host_packages is defined

I recommend to define any list as a list of dictionaries.

Manage Vercel DNS records with Ansible

Thu, 11 Feb 2021 11:41:16 +0100

At Mint System we are using Ansible extensively to configure our infrastructure and services. From setting up new hosts up to deploying a customer specific customization for an application, all is managed by Ansible. Only one piece was missing. Until recently we could not update DNS records automatically.

We use Vercel to deploy static sites and manage DNS records. Their well documented API allows you to manage any Vercel resource. Using the Ansible uri module, I have created a role that manages these ressources. Lets have a look.

This is the inventory template:

vercel_token: "{{ vault_vercel_token }}"
vercel_team_id: example-organization
vercel_dns:
  - domain: example.com
    records:
      - { name: www, type: ALIAS, value: www.example.org, state: present }
      - { name: '', type: A, value: 93.184.216.34, state: present }

You can configure domains and their DNS records.

This inventory is used by the vercel-dns task. I will now step through the task file and explain how DNS records are updated.

roles/vercel/tasks/vercel-dns.yml

- name: Get all vercel domains
  uri:
    url: "{{ vercel_api_url }}/v4/domains?teamId={{ vercel_team_id }}"
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {{ vercel_token }}"
    return_content: yes
  register: vercel_domains

- name: Set domains exist fact
  set_fact:
    domains_exist: "{{ vercel_domains.json | json_query('domains[*].name') }}"

- name: Fail if domain is not managed by vercel
  fail:
    msg: Domain is not managed by vercel
  when: item.domain not in domains_exist
  loop: "{{ vercel_dns }}"

First all managed domains are retrieved from Vercel. The second task checks wether the domain that is in the inventory is also setup in Vercel. If this is not the case, the execution will fail.

- name: Get all vercel dns records
  uri:
    url: "{{ vercel_api_url }}/v4/domains/{{ item }}/records?teamId={{ vercel_team_id }}"
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {{ vercel_token }}"
    return_content: yes
  loop: "{{ domains_exist }}"
  register: vercel_dns_records

- name: Set vercel dns simple fact
  set_fact:
    record: "{{ item.1.name }}.{{ item.0.item }}-{{ item.1.type }}"
  with_subelements:
    - "{{ vercel_dns_records.results }}"
    - json.records
  register: vercel_dns_simple

- name: Make a simple list
  set_fact:
    vercel_dns_simple: "{{ vercel_dns_simple.results | map(attribute='ansible_facts.record') | list }}"

- name: Ensure DNS entry exists
  uri:
    url: "{{ vercel_api_url }}/v2/domains/{{ item.0.domain }}/records?teamId={{ vercel_team_id }}"
    method: POST
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {{ vercel_token }}"
    body: "{{ item.1  | dict2items | rejectattr('key', 'equalto', 'state') | list | items2dict | to_json }}"
  with_subelements:
    - "{{ vercel_dns }}"
    - records
  when: ((item.1.name + "." + item.0.domain + "-" + item.1.type) not in vercel_dns_simple) and (item.1.state == 'present')
  register: response
  changed_when: response.status == 200

The next four tasks retrieve all DNS records from Vercel for each configured domain. In order to compare the inventory list and the Vercel list, the Vercel list is flattened. Ansible cannot compare objects easily, but comparing strings is fine. In the last step the task checks whether the DNS record exists and if not creates one.

- name: Set vercel dns absent fact
  set_fact:
    record: "{{ item.1.name }}.{{ item.0.domain }}-{{ item.1.type }}"
  with_subelements:
    - "{{ vercel_dns }}"
    - records
  when: item.1.state == 'absent'
  register: vercel_dns_absent

- name: Make a simple list
  set_fact:
    vercel_dns_absent: "{{ vercel_dns_absent.results | selectattr('ansible_facts.record','defined') | map(attribute='ansible_facts.record') | list }}"

- name: Ensure DNS entries to be removed
  uri:
    url: "{{ vercel_api_url }}/v2/domains/{{ item.0.item }}/records/{{ item.1.id }}?teamId={{ vercel_team_id }}"
    method: DELETE
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {{ vercel_token }}"
  with_subelements:
    - "{{ vercel_dns_records.results }}"
    - json.records
  when: (item.1.name + "." + item.0.item + "-" + item.1.type) in vercel_dns_absent
  register: response
  changed_when: response.status == 200

The last three tasks will remove a DNS record, if it has the status absent. First a simple list is created that contains all records with the absent status. Then foreach Vercel record it will check if it is in the absent list. If so the task will remove the record from Vercel.

As expected the Ansible tasks are idempotent, they will return an ok if nothing has changed. Further it is ensured that DNS records which are configured in Vercel only are not deleted or updated by acccident.

To get the lastes version of the script, have a look here: https://github.com/Mint-System/Ansible-Playbooks/blob/master/roles/vercel/tasks/vercel-dns.yml.

Pin and update Ubuntu packages with Ansible

Fri, 05 Jun 2020 20:07:00 +0200

Patching is an essential part in server maintenance. For Windows servers and clients you have WSUS to coordinate updates and security patches. But how do you patch Ubuntu, Debian, RHEL or Fedora machines? The linux distro field is very dispersed, how do you do patch all these systems? In this post I will show you how I install package updates and ensure that specific packages cannot be updated using version pinning.

The current state of patch managenent.

Enterprise derivatives

RHEL and Ubuntu are both enterprise derivates, which means that enterprise features for these distros are managed by companies and not communities.

Ubuntu is managed by Canonical and they have their Livepatch Service to apply patches.

RHEL is managed by Red Hat and they provide Satellite to apply patches, system configurations and much more.

Community derivates

Debian, Fedora and many more are distros managed by communities. If you search their name in combination with patching you end up with various blog posts, where people show how they install the kernel updates and security patches. It is obvious that patch management is executed by companies running a farm of servers and there for seek automated solutions.

However, with Ansible you can easily manage you server farm and therefore also require an advanced solution to apply patches for your hosts.

Patch Ubuntu using Ansible

First let me assure that the method for patching can also be used for other distros. For better understanding I will show how its done using Ubuntu.

In this tutorial I will show you how I install essential packages and apply system patches. For my system configurations I try to be as explicit as possible, which means I pin versions for installed software. Assume that we have a Docker installation on an Ubuntu server and want to make sure that Docker is not updated as we do not want break compatibility.

Project setup

In my Ansible project I have created the following inventory and role:

inventories/setup
├── group_vars
│   ├── all.yml
│   └── ubuntu1804.yml
├── host_vars
│   ├── server1.yml
│   └── server2.yml
└── hosts.yml

roles/update/tasks
├── main.yml
└── update-ubuntu.yml

roles/docker/tasks
├── main.yml
└── install.yml

setup.yml

The setup.yml is an Ansible playbook that installs docker and updates servers. Note that I simplified the project setup for this tutorial.

Group vars

The group vars usually contains a list of packages.

inventories/setup/group_vars/ubuntu1804.yml

docker_package: "docker-ce=5:19.03.8~3-0~ubuntu-bionic"

I will demonstrate the pin version feature using this package.

Host vars

The host vars simply tell wether this host should be updated or not.

inventories/setup/host_vars/server1.yml

update: yes

Setup role

The setup role installs docker ce and pins its version if the package string contains an =.

roles/docker/tasks/install.yml

- name: Unpin docker-ce version
  dpkg_selections:
    name: "{{ docker_package }}"
    selection: hold
  when: "'=' not in docker_package"

- name: Update apt and install docker-ce
  apt:
    update_cache: yes
    name: "{{ docker_package }}"

- name: Pin docker-ce version
  dpkg_selections:
    name: "{{ docker_package.split('=')[0] }}"
    selection: hold
  when: "'=' in docker_package"

Once a package has been pinned, it cannot be updated by the package manger.

Update role

The update role runs only if updates are enabled for the host.

update/tasks/main.yml

- name: Include update tasks
  include_tasks: update-ubuntu.yml
  when: update

The update role refreshes the repo cache and runs a dist upgrade.

update/tasks/update-ubuntu.yml

- name: Update apt-get repo and cache
  apt:
    update_cache: yes
    force_apt_get: yes
    cache_valid_time: 3600

- name: Upgrade all packages
  apt:
    upgrade: dist

- name: Check if reboot is required
  stat: 
    path: /var/run/reboot-required
  register: reboot_required

- name: Reboot system if required
  reboot:
    msg: Rebooting to complete system upgrade
    reboot_timeout: 120
  when: reboot_required.stat.exists

If a reboot is required it will do so.

Run the playbook

The Ansible playbook setup.yml supports two running modes. The default mode and the update mode (update tag is required).

- hosts: all
  become: true
  roles:
  - role: docker
    tags: docker
  - role: update
    tags: update,never

To install docker on the target host you would enter: ansible-playbook -i inventory/setup setup.yml -l server1

And to update the target host you would enter: ansible-playbook -i inventory/setup setup.yml -l server1 -t update

It is recommend the run the update command periodically.

I hope I was able to demonstrate how you can manage patches for your linux systems using Ansible. As mentioned this method can be used for other non-aptitude-enabled distros.

Backup Docker volumes with Ansible and restic

Mon, 16 Mar 2020 17:43:01 +0100

In a new assignment I’m in charge of the infrastructure for a new startup. I was given a blank canvas and decided to use Ansible and Docker from the start. Therefore I’ve setup an Ansible project containing various roles and deployment scenarios. Have a look here for details: https://github.com/Mint-System/Ansible-Playbooks. To put it simply, this project deploys open source web application as Docker containers on a target system. Currently, I am adding new features and polishing existing ones. An important role that is still missing is the backup. Having a robust and reliable backup and recovery system is key. While developing the backup system I had a few key points in mind:

Backup Docker volumes
Backup location is remote
No friction between backup and recovery
Backup tool must manage rotation policies
Docker host stores backups
Support for multiple backup clients and servers
Encrypted backups
Simple configuration

Solution

In the past I have used duplicity for my backups. I remember it as quite handy for managing backups, but complicated regarding encryption. While doing more research (aka googling duplicity alternatives), I found restic. It seems restic has become the standard for remote backups. Folks from camptocamp use it to backup their kubernetes cluster using the restic based backup tool bivac. Obviously, restic has been proven to be a worthy candidate.

Files

First I want to give you a short intro to the Ansible project. I have isolated the inventory and roles of the restic client and server deployment.

These are the relevant Ansible project files:

.
├── backup.yml
├── inventories/
│   └── backup/ # name of the inventory
│       ├── group_vars/
│       │   ├── all.yml
│       │   ├── client/
│       │   │   ├── vars.yml
│       │   │   └── vault.yml # Use ansible-vault to create this file
│       │   └── server/
│       │       ├── vars.yml
│       │       └── vault.yml  # Use ansible-vault to create this file
│       ├── host_vars/
│       │   ├── client.example.com.yml
│       │   └── server.example.com.yml
│       └── hosts.yml
└── roles/
    ├── restic-client/
    │   └── tasks/
    │   │   ├── main.yml
    │       └── install.yml
    └── restic-server/
        └── tasks/
            ├── main.yml
            └── main.yml

Inventory

Let’s have a look at the inventory.

The all.yml file contains configs which is applied to all hosts of the backup inventory.

inventories/backup/group_vars/all.yml

docker_network_name: example.com
docker_log_driver: "json-file"
docker_log_max_size: "10m"
docker_log_max_file: "3"

ansible_python_interpreter: /usr/bin/python3

For every inventory group, in this case client and server, there is a folder and a vars config file.

inventories/backup/group_vars/client/vars.yml

restic_client_package: "restic=0.8.3+ds-1"
restic_client_user: admin
restic_client_password: "{{ vault_restic_client_password }}"
restic_repo: server.example.com:8080/backup
restic_repo_password: "{{ vault_restic_repo_password }}"

Every variable that is prefixed with vault_ is defined in the vault.yml file.

inventories/backup/group_vars/server/vars.yml

# https://hub.docker.com/r/restic/rest-server
restic_server_image: restic/rest-server:0.9.7
restic_server_user: admin
restic_server_password: "{{ vault_restic_server_password }}"
restic_server_port: 8080

For host specific configurations there is a file in the host_vars folder.

inventories/backup/host_vars/client.example.com.yml

restic_backup_sets:
 - id: "odoo volume"
   type: docker-volume
   volume: odoo_data01
   tags:
  - odoo01
  - postgres01
   hour: "1"
   minute: "0"
  - id:-"postgres-volume"
   type: docker-volume
   volume: postgres_data01
   tags:
  - odoo01
  - postgres01
   hour: "1"
   minute: "0"
restic_backup_rotation:
  daily: 7
  weekly: 4
  monthly: 1

Each client has a set of backup jobs.

inventories/backup/host_vars/server.example.com.yml

restic_server_hostname: restic01
restic_server_backup_dir: /path/to/mount

The server mounts the backupfolder via bind mount.

Roles

Next we will have a look at the Ansible roles.

Restic server

The restic server receives and stores backup files.

roles/restic-server/tasks/main.yml

- name: Include install tasks
  include_tasks: install.yml
  when: restic_server_image is defined

If a restic server docker image is defined, the install tasks are included.

roles/restic-server/tasks/install.yml

- name: Install htpasswd module
  apt: 
    name: python3-passlib
    state: latest

- name: Configure user access for restic server
  htpasswd:
    path: "{{ restic_server_backup_dir }}/.htpasswd"
    name: "{{ restic_server_user }}"
    crypt_scheme: ldap_sha1
    password: "{{ restic_server_password }}"

- name: Start restic server container
  docker_container:
    name: "{{ restic_server_hostname }}"
    image: "{{ restic_server_image }}"
    volumes:
      - "{{ restic_server_backup_dir }}:/data"
    ports:
      - "{{ restic_server_port }}:8000"
    networks:
      - name: "{{ docker_network_name }}"
    log_driver: "{{ docker_log_driver }}"
    log_options:
      max-size: "{{ docker_log_max_size }}"
      max-file: "{{ docker_log_max_file }}"

The install tasks creates a .htpasswd file in the mounted directory and then starts a restic server container with configurations applied from the inventory.

As you can see the http communication is not encrypted. However, this is not a problem as restic encrypts its payload.

Restic client

The restic client runs backups and uses the server as storage.

roles/restic-client/tasks/main.yml

- name: Include install tasks
  include_tasks: install.yml
  when: restic_client_package is defined

Now comes the most interesting part. The install.yml of the restic client does a few things:

Install the restic package
Check if backup repo has been initialized and if not does so
Ensure the repo url and password are set as global environment variables
Register the backup jobs
Register the backup rotation job

roles/restic-client/tasks/install.yml

- name: Install restic
  apt:
    name: "{{ restic_client_package }}"

- name: Check if repo is initialized
  shell: restic snapshots
  environment:
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  ignore_errors: yes
  changed_when: false
  register: repo_initalized

- name: Init restic repository
  shell: restic init
  environment:
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  when: repo_initalized.failed

- name: Ensure restic environment vars exists
  lineinfile:
    dest: "/etc/environment"
    state: present
    regexp: "^{{ item.key }}="
    line: "{{ item.key }}={{ item.value}}"
  loop:
    - key: RESTIC_PASSWORD
      value: "{{ restic_repo_password }}"
    - key: RESTIC_REPOSITORY
      value: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"

- name: Register docker volume backup jobs
  cron:
    name: "Backup job {{ item.id }}"
    hour: "{{ item.hour }}"
    minute: "{{ item.minute }}"
    job: ". /etc/environment; restic backup /var/lib/docker/volumes/{{ item.volume }}/_data/ --tag {{ item.tags | join(' --tag ')}}"
  loop: "{{ restic_backup_sets }}"
  when: item.type == "docker-volume"

- name: Register backup rotation job
  cron:
    name: "Backup rotation job"
    hour: "23"
    minute: "0"
    job: ". /etc/environment; restic forget --keep-daily {{ restic_backup_rotation.daily }} --keep-weekly {{ restic_backup_rotation.weekly }} --keep-monthly {{ restic_backup_rotation.monthly }} --prune"

For every item in the restic_backup_sets var a cron job is configured. The item.type property allows you to define and select more backup types.

Deployment

This is the final section. I will show you how to deploy the roles.

Playbook

First we need an Ansible playbook.

backup.yml

- hosts: all
  become: true
  roles:
  - role: restic-server
    tags: restic-server
  - role: restic-client
    tags: restic-client

This playbook includes the role. Note that we deploy as root on the target machine.

Installation

If everything has been configured properly, we can install the server:

ansible-playbook -i inventories/backup backup.yml -l server.example.com

And the client:

ansible-playbook -i inventories/backup backup.yml -l client.example.com

Here is an example output of the client installation:

MacBuechLuft:ansible-playbooks janikvonrotz$ ansible-playbook -i inventories/backup backup.yml -t restic-client

PLAY [all] ***********************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [server.example.com]
ok: [client.example.com]

TASK [restic-client : Include install tasks] *************************************************************************************************************************************************************************************************
skipping: [server.example.com]
included: /Users/janikvonrotz/ansible-playbooks/roles/restic-client/tasks/install.yml for client.example.com

TASK [restic-client : Install restic] ********************************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Check if repo is initialized] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Init restic repository] ************************************************************************************************************************************************************************************************
skipping: [client.example.com]

TASK [restic-client : Ensure restic environment vars exists] *********************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'key': 'RESTIC_PASSWORD', 'value': '********'})
ok: [client.example.com] => (item={'key': 'RESTIC_REPOSITORY', 'value': 'rest:http://admin:'********'})@server.example.com:8080/backup'})

TASK [restic-client : Register docker volume backup jobs] ************************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'id': 'odoo volume', 'type': 'docker-volume', 'volume': 'odoo_data01', 'tags': ['odoo', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})
ok: [client.example.com] => (item={'id': 'postgres volume', 'type': 'docker-volume', 'volume': 'postgres_data01', 'tags': ['postgres', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})

TASK [restic-client : Register backup rotation job] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
client.example.com      : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
server.example.com     : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Note that the passwords are not hidden in your output. They might end up on a log aggregation server.

Manual backups

Remember that we installed the restic package on the target host and defined two important environment variables on the global scope? This helps managing backups without looking up any secrets.

I can easily browse and if necessary restore backups with personal user.

MacBuechLuft:~ janikvonrotz$ ssh client.example.com
janikvonrotz@hades:~$ restic snapshots
password is correct
ID        Date                 Host        Tags            Directory
----------------------------------------------------------------------
a8f86bf6  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/odoo_data01/_data
                                           postgres01  └── 
c0686e66  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/postgres_data01/_data
                                           postgres01  └── 
----------------------------------------------------------------------
2 snapshots

Cool isn’t it? If you have any feedback, let me know? In the near future I will add new backup types and share them in the referenced GitHub repo.

Deploy ELK stack with Ansible and Docker

Mon, 28 Oct 2019 17:18:48 +0100

This post was first published at Abilium - Blog.

Recently, we decided to setup a new monitoring service. We opted for the ELK stack. The ELK stack constists of three products:

Elasticsearch - A powerful and flexible search index.
Logstash - Log ingester, filter and forwarder.
Kibana - Dashboard for data visualization.

There exist multiple distros of the ELK stack. One of them is the Open Distro for Elasticsearch by Amazon and another one is the Elastic Stack by Elastic.

We decided to use the Elastic Stack as they provide a collection of Beats in addition. Beats are services that ship all kinds of data (Log, Metrics, Performance, Uptime) to Logstash. It makes it much easier to actually collect data of your services and forward them to Logstash.

At Abilium GmbH Docker and Kubernetes are the default way to run applications. Most times we use Jenkings and Docker Compose to build, test and deploy an application release. But we were not quite happy with docker compose as it does not support a meaningful way to configure the host system. Thus we decided use a very popular automation and configuration management tool: Ansible.

In the following paragraph we will show how you can deploy the Elastic Stack using Ansible and Docker.

Requirements

It’s assumed that you already know how to handle Ansible and are familiar with the related terms. Moreover, we assume that you know the basic idea of Docker and how it interacts with Ansible.

Regarding system requirements for Docker checkout the first section of Install Elasticsearch with Docker

Setup

Ansible in its plain form is a hierarchy of .yml files which tell what task should performed on which machine. There is no final layout when it comes to structuring these files. To get a grasp of what an Ansible project might look like, we printed the folder tree of our example project:

├── ansible.cfg # Global ansible configuration
├── elk-clean.yml # Cleanup playbook
├── elk.yml # Elastic Stack playbook
├── inventory # Inventory folder
│   ├── group_vars # Folder that contains variables grouped by environment
│   │   ├── all # Variables which apply to every environment
│   │   │   ├── vars.yml # Global configuration
│   │   │   └── vault.yml # Ansible Vault secrets
│   │   └── prod # Production environment
│   │       └── vars.yml # Environment specific folder
│   └── hosts.yml # List of hosts grouped by environment
└── roles # Roles are Ansible modules
    ├── clean # Cleanup role that removes all configurations
    │   └── tasks
    │       └── main.yml # Tasks to cleanup containers and related config
    ├── elasticsearch # The Elasticsearch role
    │   └── tasks
    │       └── main.yml  # Tasks to deploy the Elasticsearch Docker container
    ├── kibana # The Kibana role
    │   └── tasks
    │       └── main.yml # Tasks to deploy the Kibana Docker container
    ├── logstash # The Logstash role
    │   ├── handlers
    │   │   └── main.yml # Tasks which are called by other tasks via notifier
    │   ├── tasks
    │   │   └── main.yml # Tasks to deploy the Logstash Docker container
    │   └── templates # Logstash configuration files with Ansible variables
    │       ├── beats.conf
    │       └── syslog.conf
    └── nginx # The nginx module to expose the Kibana dashboard
        ├── handlers
        │   └── main.yml
        ├── tasks
        │   └── main.yml
        └── templates
            ├── nginx-certbot.conf
            └── nginx-ssl.conf

You can replicate this folder tree or checkout the single files in the follow-up sections.

Elasticsearch

This Ansible task file deploys the Elasticsearch Docker container to the target host.

roles/elasticsearch/tasks/main.yml

- name: Create elastic search volume
  docker_volume:
    name: esdata
    driver: local

- name: Start elastic search container
  docker_container:
    name: "{{ elasticsearch_hostname }}"
    image: "{{ elasticsearch_image }}"
    env:
      discovery.type: "single-node"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      xpack.security.enabled: "true"
      xpack.monitoring.collection.enabled: "true"
    volumes:
    - "esdata:/usr/share/elasticsearch/data"
    ulimits:
    - nofile:65535:65535
    networks:
    - name: "{{ network_name }}"
    state: started
    log_driver: "{{ log_driver }}"
    log_options:
      max-size: "{{ log_max_size }}"
      max-file: "{{ log_max_file }}"

- debug:
    msg: >
      Users have to be configured manually. Enable and set the password for the default users with:
      `docker exec -it {{ elasticsearch_hostname }} /bin/bash -c "elasticsearch-setup-passwords auto"`
      Then edit the `vaul.yml` file and copy the password values to the expected variable.

Kibana

Kibana is configured with env variables only.

roles/kibana/tasks/main.yml

- name: Start kibana container
  docker_container:
    name: "{{ kibana_hostname }}"
    image: "{{ kibana_image }}"
    env:
      servername: "{{ server_name }}"
      ELASTICSEARCH_HOSTS: "http://{{ elasticsearch_hostname }}:9200"
      ELASTICSEARCH_USERNAME: "kibana"
      ELASTICSEARCH_PASSWORD: "{{ kibana_password }}"
    networks:
    - name: "{{ network_name }}"
    log_driver: "{{ log_driver }}"
    log_options:
      max-size: "{{ log_max_size }}"
      max-file: "{{ log_max_file }}"

Logstash

Logstash processes data with pipelines. Each pipeline is configured in a .conf file. These files are deployed with Ansible as well. Everytime a file change occurs Ansible will restart the Logstash container.

roles/logstash/tasks/main.yml

- name: Ensure logstash pipeline conf dir exists
  file:
    path: "{{ logstash_conf_dir }}/pipeline"
    state: directory

- name: Copy logstash pipeline conf
  template: src={{ item.src }} dest={{ item.dest }}
  with_items:
    - { src: 'templates/syslog.conf', dest: '/{{ logstash_conf_dir }}/pipeline/syslog.conf' }
    - { src: 'templates/beats.conf', dest: '/{{ logstash_conf_dir }}/pipeline/beats.conf' }
  notify: Restart logstash container

- name: Start logstash container
  docker_container:
    name: "{{ logstash_hostname }}"
    image: "{{ logstash_image }}"
    env:
      XPACK_MONITORING_ELASTICSEARCH_HOSTS: "http://{{ elasticsearch_hostname }}:9200"
      XPACK_MONITORING_ENABLED: "true"
      XPACK_MONITORING_ELASTICSEARCH_USERNAME: "elastic"
      XPACK_MONITORING_ELASTICSEARCH_PASSWORD: "{{ elastic_password }}"
      PATH_CONFIG: ""
    ports:
    - 5000:5000
    - 5044:5044
    volumes:
    - "{{ logstash_conf_dir }}/pipeline:/usr/share/logstash/pipeline:ro"
    networks:
    - name: "{{ network_name }}"
    log_driver: "{{ log_driver }}"
    log_options:
      max-size: "{{ log_max_size }}"
      max-file: "{{ log_max_file }}"

roles/logstash/handlers/main.yml

Handlers are notified by tasks and will be processed at the end of a role deployment.

- name: Restart logstash container
  docker_container:
    name: "{{ logstash_hostname }}"
    restart: true

roles/logstash/templates/beats.conf

The beats pipeline forwards messages sent by Beats services to its appropriate index.

input { stdin { } }

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][beat] in ["heartbeat", "metricbeat", "filebeat"] {
    elasticsearch { 
      hosts => ["http://{{ elasticsearch_hostname }}:9200"]
      user => "elastic"
      password => "{{ elastic_password }}"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    }
  }
}

roles/logstash/templates/syslog.conf

The syslog pipeline processes syslog messages using a grok filter.

input { stdin { } }

input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}" }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
  }
}

output {
  if [type] == "syslog" {
    elasticsearch { 
      hosts => ["http://{{ elasticsearch_hostname }}:9200"]
      user => "elastic"
      password => "{{ elastic_password }}"
      index => "syslog-%{+YYYY.MM.dd}"
    }
  }
}

Nginx

It is easier to setup an Nginx proxy that secures access to the Kibana dashboard than configuring keymaterial for Kibana and setup direct access.

roles/nginx/tasks/main.yml

First this role checks if LetsEncrypt certificates have been generated for the configured domain. If this is not the case it will setup an Nginx instance whose only purpose is to complete the ACME challenge initialized by the Certbot command. Once the certificates have been generated a ssl secured Nginx instance will be deployed.

- name: Ensure nginx conf dir exists
  file:
    path: "{{ nginx_conf_dir }}"
    state: directory

- name: Check if cert files exist
  stat:
    path: "{{ certbot_conf_dir }}/live/{{ server_name }}"
  register: certbot_certs

- name: Copy nginx certbot conf
  template:
    src: templates/nginx-certbot.conf
    dest: "{{ nginx_conf_dir }}/{{ server_name }}.conf"
  when: not certbot_certs.stat.exists

- name: Start nginx container
  docker_container:
    name: ng01
    image: "{{ nginx_image }}"
    ports:
    - 80:80
    volumes:
    - "{{ nginx_conf_dir }}/:/etc/nginx/conf.d/:ro"
    - "{{ certbot_conf_dir }}/:/etc/letsencrypt/"
    - "{{ certbot_conf_dir }}/www/:/var/www/certbot/"
    log_driver: "{{ log_driver }}"
    log_options:
      max-size: "{{ log_max_size }}"
      max-file: "{{ log_max_file }}"
  when: not certbot_certs.stat.exists

- name: Wait for nginx container
  pause:
    seconds: 3
  when: not certbot_certs.stat.exists

- name: Issue certificate with certbot command
  docker_container:
    name: "{{ certbot_hostname }}"
    image: "{{ certbot_image }}"
    volumes:
    - "{{ certbot_conf_dir }}/:/etc/letsencrypt/"
    - "{{ certbot_conf_dir }}/www/:/var/www/certbot/"
    command: certonly --webroot --email {{ certbot_email }} --agree-tos --webroot-path=/var/www/certbot/ -d {{ server_name }}
  when: not certbot_certs.stat.exists

- name: Wait for certificate request
  pause:
    seconds: 3
  when: not certbot_certs.stat.exists

- name: Copy nginx ssl conf
  template:
    src: templates/nginx-ssl.conf
    dest: "{{ nginx_conf_dir }}/{{ server_name }}.conf"
  notify: Restart nginx container

- name: Copy nginx ssl param conf
  copy:
    src: "{{ item }}"
    dest: "{{ certbot_conf_dir }}/"
  with_items:
    - files/options-ssl-nginx.conf
    - files/ssl-dhparams.pem

- name: Start nginx container
  docker_container:
    name: ng01
    image: nginx:1.15-alpine
    ports:
    - 80:80
    - 443:443
    - 9200:9200
    volumes:
    - "{{ nginx_conf_dir }}/:/etc/nginx/conf.d/:ro"
    - "{{ certbot_conf_dir }}/:/etc/letsencrypt/"
    - "{{ certbot_conf_dir }}/www/:/var/www/certbot/"
    networks:
    - name: "{{ network_name }}"
    log_driver: "{{ log_driver }}"
    log_options:
      max-size: "{{ log_max_size }}"
      max-file: "{{ log_max_file }}"

roles/nginx/handlers/main.yml

The Nginx handler for config updates.

- name: Restart nginx container
  docker_container:
    name: ng01
    restart: true

roles/nginx/templates/nginx-certbot.conf

The ACME challenge Nginx config.

server {
    listen 80;
    server_name {{ server_name }};

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    } 
}

roles/nginx/templates/nginx-ssl.conf

The ssl Nginx config that secures access to the Elastic Search API and Kibana dashboard.

server {
    listen 80;
    server_name {{ server_name }};    
    
    location / {
        return 301 https://$host$request_uri;
    }

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    } 
}

# Kibana Dashboard
server {
    listen 443 ssl;
    server_name {{ server_name }};

    ssl_certificate /etc/letsencrypt/live/{{ server_name }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ server_name }}/privkey.pem;
    
    location / {
        proxy_pass http://{{ kibana_hostname }}:5601;
    }
}

# Elastic Search
server {
    listen 9200 ssl;
    server_name {{ server_name }};

    ssl_certificate /etc/letsencrypt/live/{{ server_name }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ server_name }}/privkey.pem;

    location / {
        proxy_pass http://{{ elasticsearch_hostname }}:9200;
    }
}

Note that cipher suite definitions have been removed from the example. Recommended cipher suites must be obtained by a security consultancy.

Inventory

Ansible works against multiple environments. Therefore configurations and code must be separated. Whereas roles are the code, configurations are the inventory.

inventory/hosts.yml

all:
  hosts:
  children:
    prod:
      hosts: monitoring.example.com
    int:
      hosts: monitoring-int.example.com

inventory/group_vars/all/vars.yml

These variables are set for all environments.

log_driver: "json-file"
log_max_size: "10m"
log_max_file: "3"

They configure the log rotation for the Docker daemon.

inventory/group_vars/all/vault.yml

The vault file stores secrets. It is set for all environments as well. Run the this command to edit the file:

ansible-vault edit inventory/group_vars/all/vault.yml

vault_elastic_password: password
vault_kibana_password: password

These passwords must be updated after the first depoyment.

inventory/group_vars/prod/vars.yml

If you run the Ansible deployment it will use the prod environment definitions by default. The prod environment has been configured with these variables:

server_name: monitoring.example.com
network_name: esnet

nginx_image: nginx:1.15-alpine
nginx_hostname: ng01
nginx_conf_dir: /usr/share/nginx

elasticsearch_image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
elasticsearch_hostname: es01
elasticsearch_conf_dir: /usr/share/elasticsearch

kibana_image: docker.elastic.co/kibana/kibana:7.3.2
kibana_hostname: ki01

logstash_image: docker.elastic.co/logstash/logstash:7.4.0
logstash_hostname: lo01
logstash_conf_dir: /usr/share/logstash

certbot_image: certbot/certbot
certbot_hostname: cb01
certbot_conf_dir: /usr/share/certbot
certbot_email: info@example.com

elastic_password: "{{ vault_elastic_password }}"
kibana_password: "{{ vault_kibana_password }}"

The hostname is used inside the Docker network.

Playbooks

We are getting closer to the actual Ansible deployment. Ansible playbooks connect hosts, environments and roles. They describe which role must be installed on which host.

elk.yml

This playbook installs the described roles. Note that you can use tags to filter roles for deployments.

---
- hosts: "{{ ehosts | default('prod') }}"
  become: true
  roles:
  - { role: elasticsearch, tags: ["elasticsearch"] }
  - { role: kibana, tags: ["kibana"] }
  - { role: logstash, tags: ["logstash"] }
  - { role: nginx, tags: ["nginx"] }

elk-clean.yml

The cleanup process has been separated from the installation. Cleaning up installed software requires a different order, therefore the cleanup tasks received an exclusive role.

- hosts: "{{ ehosts | default('prod') }}"
  become: true
  roles:
  - clean

Deployment

You have reached the most important section. Here we show what commands can be used to deploy the ELK stack with Ansible and Docker.

Deploy the ELK stack.

ansible-playbook -i inventory elk.yml

Deploy the Logstash role only.

ansible-playbook -i inventory elk.yml --tags logstash

Deploy the Nginx role to localhost.

ansible-playbook -i inventory elk.yml --tags nginx --extra-vars "ehosts=local"

Deploy the Nginx role to localhost with a specified user.

ansible-playbook -i inventory elk.yml --tags nginx --extra-vars "ehosts=local" -u username

Clean the ELK stack.

ansible-playbook -i inventory elk-clean.yml

Clean the Logstash role only.

ansible-playbook -i inventory elk-clean.yml --tags logstash

Demo

Let’s see what a deployment looks like in action:

That’s it! I hope you were able to follow our solution. If not let us know in the comment section.

Notes

As you might noticed there are parts of the Ansible project that have not been solved nicely. F.g. The default users have to be configured manually. We wanna address this points and show how to resolve them if there would be more ressources for engineering.

Problem: Manual setup of default users.
Solution: Create and set the password of the elastic and kibana user automatically with http requests.

Problem: Beat and syslog input services communication is not encrypted. Solution: Encrypt connection to Logstash beat and syslog input with LetsEncrypt certificates.

Problem: Document role variables. Solution: Generate a documentation based on the inventory vars and comments.

Profiling Ansible roles and tasks

Wed, 07 Mar 2018 13:11:25 +0000

Performance is critical when deploying an environment with Ansible. By default Ansible does not tell how much time elapsed for specific role or task. However, this information would be critical to identify inefficient tasks. Luckily Ansible offers an interface for callback plugins. With the help of a callback plugin one can hook into the role or task execution call. In this post I’ll show you how to configure the callback plugins for profiling roles and tasks. It is quite easy.

First create a plugin folder in your Ansible project.

mkdir callback_plugins

Step into the directory and download the plugins from the Ansible repo.

cd callback_plugins
curl https://raw.githubusercontent.com/ansible/ansible/devel/lib/ansible/plugins/callback/profile_tasks.py -o profile_tasks.py
curl https://raw.githubusercontent.com/ansible/ansible/devel/lib/ansible/plugins/callback/profile_roles.py -o profile_roles.py

To enable the plugins update the Ansible configuration file.

ansible.cfg

[defaults]
callback_whitelist = profile_tasks, profile_roles

Now whenever you run a playbook the finish report will tell how much time elapsed for each task and role.

Here is an example of the profile roles results:

=============================================================================== 
adnwildfly ------------------------------------------------------------- 54.19s
couchbase-setup -------------------------------------------------------- 52.25s
nevisproxy ------------------------------------------------------------- 38.35s
testdata-loader -------------------------------------------------------- 36.97s
keybox ----------------------------------------------------------------- 22.56s
keybox-slot ------------------------------------------------------------ 16.25s
nevisauth -------------------------------------------------------------- 15.55s
nevislogrend ----------------------------------------------------------- 15.29s
setup ------------------------------------------------------------------- 8.66s
users ------------------------------------------------------------------- 8.03s
nevissaml --------------------------------------------------------------- 3.72s
spool ------------------------------------------------------------------- 3.32s
reference-rp ------------------------------------------------------------ 1.92s
test-tools -------------------------------------------------------------- 1.01s
release ----------------------------------------------------------------- 0.62s
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
total ----------------------------------------------------------------- 278.69s

Working with Ansible - Cleanup tasks

Mon, 26 Feb 2018 11:30:13 +0000

My current project is heavily based on Ansible. We use Ansible to deploy into different environments and domains. The configuration has become quite complex. That is why we need to cleanup a server from time to time. Especially when one has to upgrade services on a server and requires to get rid of the old services first. In this post I would like to show you, how we define and run Ansible cleanup tasks in our project.

Imagine this simple directory structure:

roles/
  basic/
    tasks/
      main.yml
      clean.yml

With a very basic task:

main.yml

- name: install packages
  yum: name={{ item }} state=present
  with_items:
    - nano
    - git
    - vim

Now there is a scenario where we need to get rid of these packages before running the main task. For this reason with have the cleanup task:

clean.yml

- name: uninstall packages
  yum: name={{ item }} state=absent
  with_items:
    - nano
    - git
    - vim

It simply does the opposite of the main task.

Now how do we invoke the cleanup task? Well just include the task conditionally like this:

main.yml

- include: clean.yml
  when: clean

- name: install packages
...

Now whenever the main task is executed it will include the cleanup task if the variable clean is true. To run a reinstallation use the command: ansible-playbook _PLAYBOOK_.yml -e clean=true.

It is also possible to run the cleanup task only by using tags.

- include: clean.yml
  when: clean
  tags: clean

- name: install packages
...

Remember when you set a tag the tasks only using the tag are executed. Run this command to uninstall only: ansible-playbook _PLAYBOOK_.yml -e clean=true -t clean.

Of course you should avoid having cleanup task in the first place. Defining a rollback state is not very Ansible like.