Janik von Rotz

7 min read

Backup Docker volumes with Ansible and restic

In a new assignment I’m in charge of the infrastructure for a new startup. I was given a blank canvas and decided to use Ansible and Docker from the start. Therefore I’ve setup an Ansible project containing various roles and deployment scenarios. Have a look here for details: https://github.com/Mint-System/Ansible-Playbooks. To put it simply, this project deploys open source web application as Docker containers on a target system. Currently, I am adding new features and polishing existing ones. An important role that is still missing is the backup. Having a robust and reliable backup and recovery system is key. While developing the backup system I had a few key points in mind:


In the past I have used duplicity for my backups. I remember it as quite handy for managing backups, but complicated regarding encryption. While doing more research (aka googling duplicity alternatives), I found restic. It seems restic has become the standard for remote backups. Folks from camptocamp use it to backup their kubernetes cluster using the restic based backup tool bivac. Obviously, restic has been proven to be a worthy candidate.


First I want to give you a short intro to the Ansible project. I have isolated the inventory and roles of the restic client and server deployment.

These are the relevant Ansible project files:

├── backup.yml
├── inventories/
│   └── backup/ # name of the inventory
│       ├── group_vars/
│       │   ├── all.yml
│       │   ├── client/
│       │   │   ├── vars.yml
│       │   │   └── vault.yml # Use ansible-vault to create this file
│       │   └── server/
│       │       ├── vars.yml
│       │       └── vault.yml  # Use ansible-vault to create this file
│       ├── host_vars/
│       │   ├── client.example.com.yml
│       │   └── server.example.com.yml
│       └── hosts.yml
└── roles/
    ├── restic-client/
    │   └── tasks/
    │   │   ├── main.yml
    │       └── install.yml
    └── restic-server/
        └── tasks/
            ├── main.yml
            └── main.yml


Let’s have a look at the inventory.

The all.yml file contains configs which is applied to all hosts of the backup inventory.


docker_network_name: example.com
docker_log_driver: "json-file"
docker_log_max_size: "10m"
docker_log_max_file: "3"

ansible_python_interpreter: /usr/bin/python3

For every inventory group, in this case client and server, there is a folder and a vars config file.


restic_client_package: "restic=0.8.3+ds-1"
restic_client_user: admin
restic_client_password: "{{ vault_restic_client_password }}"
restic_repo: server.example.com:8080/backup
restic_repo_password: "{{ vault_restic_repo_password }}"

Every variable that is prefixed with vault_ is defined in the vault.yml file.


# https://hub.docker.com/r/restic/rest-server
restic_server_image: restic/rest-server:0.9.7
restic_server_user: admin
restic_server_password: "{{ vault_restic_server_password }}"
restic_server_port: 8080

For host specific configurations there is a file in the host_vars folder.


 - id: "odoo volume"
   type: docker-volume
   volume: odoo_data01
    - odoo01
    - postgres01
   hour: "1"
   minute: "0"
 - id: "postgres volume"
   type: docker-volume
   volume: postgres_data01
    - odoo01
    - postgres01
   hour: "1"
   minute: "0"
  daily: 7
  weekly: 4
  monthly: 1

Each client has a set of backup jobs.


restic_server_hostname: restic01
restic_server_backup_dir: /path/to/mount

The server mounts the backupfolder via bind mount.


Next we will have a look at the Ansible roles.

Restic server

The restic server receives and stores backup files.


- name: Include install tasks
  include_tasks: install.yml
  when: restic_server_image is defined

If a restic server docker image is defined, the install tasks are included.


- name: Install htpasswd module
    name: python3-passlib
    state: latest

- name: Configure user access for restic server
    path: "{{ restic_server_backup_dir }}/.htpasswd"
    name: "{{ restic_server_user }}"
    crypt_scheme: ldap_sha1
    password: "{{ restic_server_password }}"

- name: Start restic server container
    name: "{{ restic_server_hostname }}"
    image: "{{ restic_server_image }}"
      - "{{ restic_server_backup_dir }}:/data"
      - "{{ restic_server_port }}:8000"
      - name: "{{ docker_network_name }}"
    log_driver: "{{ docker_log_driver }}"
      max-size: "{{ docker_log_max_size }}"
      max-file: "{{ docker_log_max_file }}"

The install tasks creates a .htpasswd file in the mounted directory and then starts a restic server container with configurations applied from the inventory.

As you can see the http communication is not encrypted. However, this is not a problem as restic encrypts its payload.

Restic client

The restic client runs backups and uses the server as storage.


- name: Include install tasks
  include_tasks: install.yml
  when: restic_client_package is defined

Now comes the most interesting part. The install.yml of the restic client does a few things:


- name: Install restic
    name: "{{ restic_client_package }}"

- name: Check if repo is initialized
  shell: restic snapshots
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  ignore_errors: yes
  changed_when: false
  register: repo_initalized

- name: Init restic repository
  shell: restic init
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  when: repo_initalized.failed

- name: Ensure restic environment vars exists
    dest: "/etc/environment"
    state: present
    regexp: "^{{ item.key }}="
    line: "{{ item.key }}={{ item.value}}"
      value: "{{ restic_repo_password }}"
      value: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"

- name: Register docker volume backup jobs
    name: "Backup job {{ item.id }}"
    hour: "{{ item.hour }}"
    minute: "{{ item.minute }}"
    job: ". /etc/environment; restic backup /var/lib/docker/volumes/{{ item.volume }}/_data/ --tag {{ item.tags | join(' --tag ')}}"
  loop: "{{ restic_backup_sets }}"
  when: item.type == "docker-volume"

- name: Register backup rotation job
    name: "Backup rotation job"
    hour: "23"
    minute: "0"
    job: ". /etc/environment; restic forget --keep-daily {{ restic_backup_rotation.daily }} --keep-weekly {{ restic_backup_rotation.weekly }} --keep-monthly {{ restic_backup_rotation.monthly }} --prune"

For every item in the restic_backup_sets var a cron job is configured. The item.type property allows you to define and select more backup types.


This is the final section. I will show you how to deploy the roles.


First we need an Ansible playbook.


- hosts: all
  become: true
  - role: restic-server
    tags: restic-server
  - role: restic-client
    tags: restic-client

This playbook includes the role. Note that we deploy as root on the target machine.


If everything has been configured properly, we can install the server:

ansible-playbook -i inventories/backup backup.yml -l server.example.com

And the client:

ansible-playbook -i inventories/backup backup.yml -l client.example.com

Here is an example output of the client installation:

MacBuechLuft:ansible-playbooks janikvonrotz$ ansible-playbook -i inventories/backup backup.yml -t restic-client

PLAY [all] ***********************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [server.example.com]
ok: [client.example.com]

TASK [restic-client : Include install tasks] *************************************************************************************************************************************************************************************************
skipping: [server.example.com]
included: /Users/janikvonrotz/ansible-playbooks/roles/restic-client/tasks/install.yml for client.example.com

TASK [restic-client : Install restic] ********************************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Check if repo is initialized] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Init restic repository] ************************************************************************************************************************************************************************************************
skipping: [client.example.com]

TASK [restic-client : Ensure restic environment vars exists] *********************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'key': 'RESTIC_PASSWORD', 'value': '********'})
ok: [client.example.com] => (item={'key': 'RESTIC_REPOSITORY', 'value': 'rest:http://admin:'********'})@server.example.com:8080/backup'})

TASK [restic-client : Register docker volume backup jobs] ************************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'id': 'odoo volume', 'type': 'docker-volume', 'volume': 'odoo_data01', 'tags': ['odoo', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})
ok: [client.example.com] => (item={'id': 'postgres volume', 'type': 'docker-volume', 'volume': 'postgres_data01', 'tags': ['postgres', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})

TASK [restic-client : Register backup rotation job] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
client.example.com      : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
server.example.com     : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0  

Note that the passwords are not hidden in your output. They might end up on a log aggregation server.

Manual backups

Remember that we installed the restic package on the target host and defined two important environment variables on the global scope? This helps managing backups without looking up any secrets.

I can easily browse and if necessary restore backups with personal user.

MacBuechLuft:~ janikvonrotz$ ssh client.example.com
janikvonrotz@hades:~$ restic snapshots
password is correct
ID        Date                 Host        Tags            Directory
a8f86bf6  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/odoo_data01/_data
                                           postgres01  └── 
c0686e66  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/postgres_data01/_data
                                           postgres01  └── 
2 snapshots

Cool isn’t it? If you have any feedback, let me know? In the near future I will add new backup types and share them in the referenced GitHub repo.

Categories: DevOps
Tags: backup , ansible , restic
Improve this page
Show statistic for this page