Backup on Janik von Rotz

Backup Docker volumes with this script

Tue, 18 Aug 2020 21:38:34 +0200

The official Docker documentation has section about container backup. This section tells you how to backup an attached volume and the command to do so is pretty elaborate. For automating backups you don’t want to script parametrized commands such as this one. A simple script will do the job.

Here is a script that I use to backup all mounted volumes from a running container.

docker-volume-backup

#!/bin/bash

# Exit script if command fails
set -e

# Display Help
Help() {
  echo
  echo "docker-volume-backup"
  echo "####################"
  echo
  echo "Description: Backup docker volumes."
  echo "Syntax: docker-volume-backup [-v|-a|-o|-c|help]"
  echo "Example: docker-volume-backup -v postgres_data01 -o /tmp -c postgres01"
  echo "options:"
  echo "  -v    Comma-separated list of volume names."
  echo "  -a    Backup all volumes."
  echo "  -o    Output directory. Defaults to '/var/tmp'"
  echo "  -c    Docker container name."
  echo "  help  Show docker-volume-backup manual."
  echo
}

# Show help and exit
if [[ $1 == 'help' ]]; then
    Help
    exit
fi

# Process params
while getopts ":a :c: :v: :o:" opt; do
  case $opt in
    a) ALL='true'
    ;;
    c) CONTAINER="$OPTARG"
    ;;
    v) VOLUMES="$OPTARG"
    ;;
    o) DIR="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    Help
    exit;;
  esac
done

# Fallback to environment vars and default values
: ${DIR:='/var/tmp'}

# Verify variables
[[ -z "$VOLUMES" ]] && [[ -z "$ALL" ]] && { echo "Parameter -v or -a|volumes or all must be set" ; exit 1; }
[[ -z "$DIR" ]] && { echo "Parameter -d|dir is empty" ; exit 1; }
[[ -z "$CONTAINER" ]] && { echo "Parameter -c|container is empty" ; exit 1; }

if $ALL ; then
  # Get all volumes from docker container
  VOLUME_LIST=($(docker inspect -f '{{ range .Mounts }}{{ .Name }} {{ end }}' $CONTAINER))

  # Concate volume list
  printf -v VOLUMES '%s,' "${VOLUME_LIST[@]}"
  VOLUMES="${VOLUMES%,}"
fi

if [[ ! -z $VOLUMES ]] ; then
  # Split volumes param values
  VOLUME_LIST=($(echo $VOLUMES | tr "," "\n"))
fi

# Create backup folder
mkdir -p ${DIR}/${CONTAINER}

# Cleanup backup folder
rm -rf ${DIR}/${CONTAINER}/*

# Create dump with docker for each database
for VOLUME in "${VOLUME_LIST[@]}"
do
  echo "Run backup for Docker volume $VOLUME"
  docker run --rm -v $VOLUME:/_data  -v ${DIR/./$(PWD)}/${CONTAINER}:/backup ubuntu tar cf /backup/$VOLUME.tar /_data
done

# Notify if backup has finished
echo "The Docker volume backup has finished: ${DIR}/${CONTAINER}/{${VOLUMES}}.tar"

To get the latest version of this see the Ansible Playbooks projects.

Restore Odoo database with this script

Tue, 19 May 2020 10:29:27 +0200

Odoo’s database manager provides an simple interface to backup and restore an Odoo database (aka tenant). This interface can be used to run scripted restores. In the same manner as the Odoo backup script I’ve created a restore script.

Copy the following script to an Odoo server to have the restore command at your hand.

/usr/local/bin/odoo-restore

#!/bin/bash

# Exit script if command fails
# -u stops the script on unset variables
# -e stops the script on errors
# -o pipefail stops the script if a pipe fails
set -e


# Get script name
SCRIPT=$(basename "$0")

# Display Help
Help() {
  echo
  echo "$SCRIPT"
  echo
  echo "Description: Restore odoo database."
  echo "Syntax: $SCRIPT [-p|-d|-f|-h|-r|help]"
  echo "Example: $SCRIPT -p secret -d odoo -f /tmp/odoo.zip -h https://odoo.example.org"
  echo "options:"
  echo "  -p    Odoo master password. Defaults to \$ODOO_MASTER_PASSWORD env var and 'admin'."
  echo "  -d    Database name. Defaults to filename."
  echo "  -f    Odoo database backup file. Defaults to '/var/tmp/odoo.zip'"
  echo "  -h    Odoo host. Defaults to 'http://localhost:8069'"
  echo "  -r    Replace existing database."  
  echo "  help  Show $SCRIPT manual."
  echo
}

# Show help and exit
if [[ $1 == 'help' ]]; then
    Help
    exit
fi

# Initialise option flag with a false value
REPLACE='false'

# Process params
while getopts ":r :p: :d: :f: :h:" opt; do
  case $opt in
    r) REPLACE='true'
    ;;
    h) ODOO_HOST="$OPTARG"
    ;;
    p) PASSWORD="$OPTARG"
    ;;
    d) DATABASE="$OPTARG"
    ;;
    f) FILE="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    Help
    ;;
  esac
done

# Fallback to environment vars and default values
: ${PASSWORD:=${ODOO_MASTER_PASSWORD:='admin'}}
: ${FILE:='/var/tmp/odoo.zip'}
: ${ODOO_HOST:='http://localhost:8069'}
FILENAME=$(basename -- "$FILE")
: ${DATABASE:="${FILENAME%%.*}"}

# Verify variables
[[ "$ODOO_HOST" == http* ]]  || { echo "Parameter -h|host must start with 'http/s'" ; exit 1; }
[[ "${FILENAME##*.}" != "zip" ]] && { echo "Parameter -f|filename must end with '.zip'" ; exit 1; }

# Validate zip file
unzip -q -t $FILE

if $REPLACE; then
  echo "Deleting Odoo database $DATABASE ..."

  curl \
    --silent \
    -F "master_pwd=${PASSWORD}" \
    -F "name=${DATABASE}" \
    ${ODOO_HOST%/}/web/database/drop | grep -q -E 'Internal Server Error|Redirecting...'
fi

# Start restore
echo "Requesting restore for Odoo database $DATABASE ..."

# Request restore with curl
CURL=$(curl \
  -F "master_pwd=$PASSWORD" \
  -F "name=$DATABASE" \
  -F backup_file=@$FILE \
  -F 'copy=true' \
  "${ODOO_HOST%/}/web/database/restore")
  
(echo $CURL | grep -q 'Redirecting...') || (echo "The restore failed:"; echo $CURL | grep error; exit 1)

echo "The restore for Odoo database $DATABASE has finished."

The Odoo master password can be declared in /etc/environments. Ensure that the script is executable.

Automate Odoo backups with this script

Thu, 02 Apr 2020 11:38:06 +0200

Odoo’s database manager provides an simple interface to backup an odoo database (tenant). This interface can be used to run automated backups. I have created a script to easily request odoo backup archives. The script works like every other command line tool.

Create the following script on a server that is running an Odoo instance.

/usr/local/bin/odoo-backup

#!/bin/bash

# Exit script if command fails
# -u stops the script on unset variables
# -e stops the script on errors
# -o pipefail stops the script if a pipe fails
set -e

# Get script name
SCRIPT=$(basename "$0")

# Display Help
Help() {
  echo
  echo "$SCRIPT"
  echo
  echo "Description: Backup odoo database."
  echo "Syntax: $SCRIPT [-p|-d|-o|-h|help]"
  echo "Example: $SCRIPT -p secret -d odoo -o /tmp -h https://odoo.example.com"
  echo "options:"
  echo "  -p    Odoo master password. Defaults to \$ODOO_MASTER_PASSWORD env var."
  echo "  -d    Database name."
  echo "  -o    Output directory with or without name. Defaults to '/var/tmp'"
  echo "  -h    Odoo host. Defaults to 'http://localhost:8069'"
  echo "  help  Show $SCRIPT manual."
  echo
}

# Show help and exit
if [[ $1 == 'help' ]]; then
    Help
    exit
fi

# Process params
while getopts ":p: :d: :o: :h:" opt; do
  case $opt in
    h) ODOO_HOST="$OPTARG"
    ;;
    p) PASSWORD="$OPTARG"
    ;;
    d) DATABASE="$OPTARG"
    ;;
    o) OUTPUT="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    Help
    exit;;
  esac
done

# Fallback to environment vars and default values
: ${PASSWORD:=${ODOO_MASTER_PASSWORD:='admin'}}
: ${OUTPUT:='/var/tmp'}
: ${ODOO_HOST:='http://localhost:8069'}

# Verify variables
[[ -z "$DATABASE" ]] && { echo "Parameter -d|database is empty" ; exit 1; }
[[ -z "$OUTPUT" ]] && { echo "Parameter -d|dir is empty" ; exit 1; }
[[ -z "$ODOO_HOST" ]] && { echo "Parameter -h|host is empty" ; exit 1; }

# Check if dir var is file or folder
if [ "${OUTPUT: -4}" = ".zip" ];then
  DIR=`dirname "$OUTPUT"`
  FILE=`basename "$OUTPUT"`
else
  DIR=$OUTPUT
fi

# Ensure output directory exists
mkdir -p $DIR

# Ensure file name is set
if [ -z "$FILE" ];then
  FILE="$DATABASE.zip"
fi

# Set file path
FILEPATH="${DIR}/${FILE}"

echo "Backup database $DATABASE to $FILEPATH"

# Request backup with curl
curl -X POST \
  -F "master_pwd=$PASSWORD" \
  -F "name=$DATABASE" \
  -F "backup_format=zip" \
  -o "$FILEPATH" \
  "$ODOO_HOST/web/database/backup"

# Grep error if is html response
FILETYPE=$(file --mime-type -b "$FILEPATH")
if [[ "$FILETYPE" == 'text/html' ]]; then
  grep error "$FILEPATH"
fi

# Validate zip file
unzip -q -t "$FILEPATH"

# Notify if backup has finished
echo "The Odoo backup has finished: $FILEPATH"

Ensure the script is executable.

$ sudo chmod +x /usr/local/bin/odoo-backup.

Create an odoo backup for your database like this:

$ odoo-backup -d nameofyourdatabase -h https://odoo.example.com -p secretmasterpassword

Assuming you want to automate backups you can create a cron job and provide the master password via environment variable. Here is an example of a cron job:

$ sudo crontab -l
#Ansible: Backup job odoo backup erp
30 1 * * * . /etc/environment; odoo-backup -d erp; restic backup /var/tmp/erp.zip --tag odoo --tag odoo01

The Odoo master password can be declared in /etc/environments.

This example is part of my Ansible Playbook project

Backup Docker volumes with Ansible and restic

Mon, 16 Mar 2020 17:43:01 +0100

In a new assignment I’m in charge of the infrastructure for a new startup. I was given a blank canvas and decided to use Ansible and Docker from the start. Therefore I’ve setup an Ansible project containing various roles and deployment scenarios. Have a look here for details: https://github.com/Mint-System/Ansible-Playbooks. To put it simply, this project deploys open source web application as Docker containers on a target system. Currently, I am adding new features and polishing existing ones. An important role that is still missing is the backup. Having a robust and reliable backup and recovery system is key. While developing the backup system I had a few key points in mind:

Backup Docker volumes
Backup location is remote
No friction between backup and recovery
Backup tool must manage rotation policies
Docker host stores backups
Support for multiple backup clients and servers
Encrypted backups
Simple configuration

Solution

In the past I have used duplicity for my backups. I remember it as quite handy for managing backups, but complicated regarding encryption. While doing more research (aka googling duplicity alternatives), I found restic. It seems restic has become the standard for remote backups. Folks from camptocamp use it to backup their kubernetes cluster using the restic based backup tool bivac. Obviously, restic has been proven to be a worthy candidate.

Files

First I want to give you a short intro to the Ansible project. I have isolated the inventory and roles of the restic client and server deployment.

These are the relevant Ansible project files:

.
├── backup.yml
├── inventories/
│   └── backup/ # name of the inventory
│       ├── group_vars/
│       │   ├── all.yml
│       │   ├── client/
│       │   │   ├── vars.yml
│       │   │   └── vault.yml # Use ansible-vault to create this file
│       │   └── server/
│       │       ├── vars.yml
│       │       └── vault.yml  # Use ansible-vault to create this file
│       ├── host_vars/
│       │   ├── client.example.com.yml
│       │   └── server.example.com.yml
│       └── hosts.yml
└── roles/
    ├── restic-client/
    │   └── tasks/
    │   │   ├── main.yml
    │       └── install.yml
    └── restic-server/
        └── tasks/
            ├── main.yml
            └── main.yml

Inventory

Let’s have a look at the inventory.

The all.yml file contains configs which is applied to all hosts of the backup inventory.

inventories/backup/group_vars/all.yml

docker_network_name: example.com
docker_log_driver: "json-file"
docker_log_max_size: "10m"
docker_log_max_file: "3"

ansible_python_interpreter: /usr/bin/python3

For every inventory group, in this case client and server, there is a folder and a vars config file.

inventories/backup/group_vars/client/vars.yml

restic_client_package: "restic=0.8.3+ds-1"
restic_client_user: admin
restic_client_password: "{{ vault_restic_client_password }}"
restic_repo: server.example.com:8080/backup
restic_repo_password: "{{ vault_restic_repo_password }}"

Every variable that is prefixed with vault_ is defined in the vault.yml file.

inventories/backup/group_vars/server/vars.yml

# https://hub.docker.com/r/restic/rest-server
restic_server_image: restic/rest-server:0.9.7
restic_server_user: admin
restic_server_password: "{{ vault_restic_server_password }}"
restic_server_port: 8080

For host specific configurations there is a file in the host_vars folder.

inventories/backup/host_vars/client.example.com.yml

restic_backup_sets:
 - id: "odoo volume"
   type: docker-volume
   volume: odoo_data01
   tags:
  - odoo01
  - postgres01
   hour: "1"
   minute: "0"
  - id:-"postgres-volume"
   type: docker-volume
   volume: postgres_data01
   tags:
  - odoo01
  - postgres01
   hour: "1"
   minute: "0"
restic_backup_rotation:
  daily: 7
  weekly: 4
  monthly: 1

Each client has a set of backup jobs.

inventories/backup/host_vars/server.example.com.yml

restic_server_hostname: restic01
restic_server_backup_dir: /path/to/mount

The server mounts the backupfolder via bind mount.

Roles

Next we will have a look at the Ansible roles.

Restic server

The restic server receives and stores backup files.

roles/restic-server/tasks/main.yml

- name: Include install tasks
  include_tasks: install.yml
  when: restic_server_image is defined

If a restic server docker image is defined, the install tasks are included.

roles/restic-server/tasks/install.yml

- name: Install htpasswd module
  apt: 
    name: python3-passlib
    state: latest

- name: Configure user access for restic server
  htpasswd:
    path: "{{ restic_server_backup_dir }}/.htpasswd"
    name: "{{ restic_server_user }}"
    crypt_scheme: ldap_sha1
    password: "{{ restic_server_password }}"

- name: Start restic server container
  docker_container:
    name: "{{ restic_server_hostname }}"
    image: "{{ restic_server_image }}"
    volumes:
      - "{{ restic_server_backup_dir }}:/data"
    ports:
      - "{{ restic_server_port }}:8000"
    networks:
      - name: "{{ docker_network_name }}"
    log_driver: "{{ docker_log_driver }}"
    log_options:
      max-size: "{{ docker_log_max_size }}"
      max-file: "{{ docker_log_max_file }}"

The install tasks creates a .htpasswd file in the mounted directory and then starts a restic server container with configurations applied from the inventory.

As you can see the http communication is not encrypted. However, this is not a problem as restic encrypts its payload.

Restic client

The restic client runs backups and uses the server as storage.

roles/restic-client/tasks/main.yml

- name: Include install tasks
  include_tasks: install.yml
  when: restic_client_package is defined

Now comes the most interesting part. The install.yml of the restic client does a few things:

Install the restic package
Check if backup repo has been initialized and if not does so
Ensure the repo url and password are set as global environment variables
Register the backup jobs
Register the backup rotation job

roles/restic-client/tasks/install.yml

- name: Install restic
  apt:
    name: "{{ restic_client_package }}"

- name: Check if repo is initialized
  shell: restic snapshots
  environment:
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  ignore_errors: yes
  changed_when: false
  register: repo_initalized

- name: Init restic repository
  shell: restic init
  environment:
    RESTIC_PASSWORD: "{{ restic_repo_password }}"
    RESTIC_REPOSITORY: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"
  when: repo_initalized.failed

- name: Ensure restic environment vars exists
  lineinfile:
    dest: "/etc/environment"
    state: present
    regexp: "^{{ item.key }}="
    line: "{{ item.key }}={{ item.value}}"
  loop:
    - key: RESTIC_PASSWORD
      value: "{{ restic_repo_password }}"
    - key: RESTIC_REPOSITORY
      value: "rest:http://{{ restic_client_user }}:{{ restic_client_password }}@{{ restic_repo }}"

- name: Register docker volume backup jobs
  cron:
    name: "Backup job {{ item.id }}"
    hour: "{{ item.hour }}"
    minute: "{{ item.minute }}"
    job: ". /etc/environment; restic backup /var/lib/docker/volumes/{{ item.volume }}/_data/ --tag {{ item.tags | join(' --tag ')}}"
  loop: "{{ restic_backup_sets }}"
  when: item.type == "docker-volume"

- name: Register backup rotation job
  cron:
    name: "Backup rotation job"
    hour: "23"
    minute: "0"
    job: ". /etc/environment; restic forget --keep-daily {{ restic_backup_rotation.daily }} --keep-weekly {{ restic_backup_rotation.weekly }} --keep-monthly {{ restic_backup_rotation.monthly }} --prune"

For every item in the restic_backup_sets var a cron job is configured. The item.type property allows you to define and select more backup types.

Deployment

This is the final section. I will show you how to deploy the roles.

Playbook

First we need an Ansible playbook.

backup.yml

- hosts: all
  become: true
  roles:
  - role: restic-server
    tags: restic-server
  - role: restic-client
    tags: restic-client

This playbook includes the role. Note that we deploy as root on the target machine.

Installation

If everything has been configured properly, we can install the server:

ansible-playbook -i inventories/backup backup.yml -l server.example.com

And the client:

ansible-playbook -i inventories/backup backup.yml -l client.example.com

Here is an example output of the client installation:

MacBuechLuft:ansible-playbooks janikvonrotz$ ansible-playbook -i inventories/backup backup.yml -t restic-client

PLAY [all] ***********************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [server.example.com]
ok: [client.example.com]

TASK [restic-client : Include install tasks] *************************************************************************************************************************************************************************************************
skipping: [server.example.com]
included: /Users/janikvonrotz/ansible-playbooks/roles/restic-client/tasks/install.yml for client.example.com

TASK [restic-client : Install restic] ********************************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Check if repo is initialized] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

TASK [restic-client : Init restic repository] ************************************************************************************************************************************************************************************************
skipping: [client.example.com]

TASK [restic-client : Ensure restic environment vars exists] *********************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'key': 'RESTIC_PASSWORD', 'value': '********'})
ok: [client.example.com] => (item={'key': 'RESTIC_REPOSITORY', 'value': 'rest:http://admin:'********'})@server.example.com:8080/backup'})

TASK [restic-client : Register docker volume backup jobs] ************************************************************************************************************************************************************************************
ok: [client.example.com] => (item={'id': 'odoo volume', 'type': 'docker-volume', 'volume': 'odoo_data01', 'tags': ['odoo', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})
ok: [client.example.com] => (item={'id': 'postgres volume', 'type': 'docker-volume', 'volume': 'postgres_data01', 'tags': ['postgres', 'odoo02', 'postgres03'], 'hour': '1', 'minute': '0'})

TASK [restic-client : Register backup rotation job] ******************************************************************************************************************************************************************************************
ok: [client.example.com]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
client.example.com      : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
server.example.com     : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Note that the passwords are not hidden in your output. They might end up on a log aggregation server.

Manual backups

Remember that we installed the restic package on the target host and defined two important environment variables on the global scope? This helps managing backups without looking up any secrets.

I can easily browse and if necessary restore backups with personal user.

MacBuechLuft:~ janikvonrotz$ ssh client.example.com
janikvonrotz@hades:~$ restic snapshots
password is correct
ID        Date                 Host        Tags            Directory
----------------------------------------------------------------------
a8f86bf6  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/odoo_data01/_data
                                           postgres01  └── 
c0686e66  2020-03-23 14:07:02  hades       odoo01      ┌── /var/lib/docker/volumes/postgres_data01/_data
                                           postgres01  └── 
----------------------------------------------------------------------
2 snapshots

Cool isn’t it? If you have any feedback, let me know? In the near future I will add new backup types and share them in the referenced GitHub repo.

Backup server installations

Mon, 08 Dec 2014 10:04:07 +0000

This post is part of my Your own Virtual Private Server hosting solution project.
Get the latest version of this article here: https://gist.github.com/676d3ab77d9d48cba73e.

Introduction

This article shows a list of the most important files and folders of your VPC installations, you definitely should backup. It assumes you’ll run a daily backup for this ressources.

Requirements

Ubuntu server
Supported installations (see below)

Instructions

As long new versions of your installations are released frequently, it’s recommanded to backup data and configuration files only.

Ubuntu

duplicity-backup

/etc/duplicity-backup/*

Koken

/var/www/<koken>

MySQL

If you have automysqlbackup running.

/var/backups/mysql/latest/*

Nginx

/etc/nginx/conf.d/*

Postfix

/etc/postfix/virtual
/etc/postfix/main.cf

Piwik

/var/www/<piwik>/config/*

WordPress

/var/www/<wordpress>/wp-config.php
/var/www/<wordpress>/wp-content/*

Backup your OneNote files

Fri, 17 Oct 2014 11:01:23 +0000

If you’re using OneDrive to sync your OneNote files, all OneNote data is stored online. There’s no “real” file containing your notes on your computer. However by default OneNote backs up your Notebook. But you might not be shure where that is?

In case you want to backup these files to an external drive you can simply copy the content of the folder: C:\Users\<user>\AppData\Local\Microsoft\OneNote\15.0\backup (this is where OneNote stores it’s backups by default).

The other option is to move the backup folder to a directory of your preference:

With OneNote open, go to “FILE” > “Options” > “Save & Backup.”
In the “Save” section, click on the file location next to “Backup Folder” > Click [Modify…].
Navigate to the file location that you would like to use as the backup location for OneNote.
Click [Select].

Unattended Encrypted Incremental Backup to Amazon S3

Fri, 25 Apr 2014 07:28:09 +0000

This post is part of my Your own Virtual Private Server hosting solution project.
Get the latest version of this article here: https://gist.github.com/9410478.

Introduction

For this task we are going to configure a duplicity script wrapper. Unregarded of the installation instructions it’s expected that you have already signed up for an Amazon account and know how to use their services.

Requirements

Installation

Clone the GitHub project.

cd /usr/local/src
sudo git clone https://github.com/zertrin/duplicity-backup.git

Copy the configuration file.

cd /usr/local/src/duplicity-backup
sudo mkdir /etc/duplicity-backup
sudo cp duplicity-backup.conf.example /etc/duplicity-backup/duplicity-backup.conf

Edit the configuration file.

sudo vi /etc/duplicity-backup/duplicity-backup.conf

Update the AWS credentials.

AWS_ACCESS_KEY_ID="[aws key id]"
AWS_SECRET_ACCESS_KEY="[aws access key]"

Update the gpg encryption settings.

PASSPHRASE="[gpg passphrase]"
GPG_ENC_KEY="[gpg key id]"
GPG_SIGN_KEY="[gpg key id]"

In case you can’t remember the gpg key id use the GnuPG tool.

gpg -k

Where the gpg key id is displayed in the line pub 2048R/>>C58886FB<< 2014-03-14

Set the backup start directory.

ROOT="/var"

Set the S3 destination bucket.

DEST="s3+http://[bucket name]/[backup-folder]/"

Define which folder should be included in the backup.

INCLIST=(  "/var/backups/mysql/latest" 
           "/var/www/<wordpress>/wp-content" 
        )

Define which folders inside the include folders should be ignored.

EXCLIST=(  "/var/www/[wordpress]/wp-content/backupwordpress*" )

Update the logging settings.

LOGDIR="/var/log/duplicity-backup/"
LOG_FILE="duplicity-`date +%Y-%m-%d_%H-%M`.txt"
LOG_FILE_OWNER="[group]:[user]"

Don’t forget to create the log folder.

sudo mkdir /var/log/duplicity-backup

Run the script to check wether it works or not.

sudo /usr/local/src/duplicity-backup/duplicity-backup.sh -c /etc/duplicity-backup/duplicity-backup.conf -b

To debug errors you can add the parameter -d, adjust the verbosity level with -v[1-9] and check the duplicity log.

In case you’ll get the error Import of duplicity.backends.giobackend Failed: No module named gio or BackendException: Could not initialize backend: No module named paramiko you have to install some additional pyhton packages.

sudo apt-get install python-paramiko python-gobject-2

In addition if you want to use gdocs as a destination you have to install the according python libarary.

sudo apt-get install pyhton-gdata

You can list the current backup with the parameter --list-current-files.

sudo /usr/local/src/duplicity-backup/duplicity-backup.sh -c /etc/duplicity-backup/duplicity-backup.conf --list-current-files

Get further help for the backup script by running the script without parameters.

 sudo /usr/local/src/duplicity-backup/duplicity-backup.sh

scheduling the job is easily done by adding a new line to the cron configuration file.

sudo vi /etc/crontab

Add the backup schedule command. The following example is executed daily a 7 o’clock.

00 7    * * *   [user]   sudo /usr/local/src/duplicity-backup/duplicity-backup.sh -c /etc/duplicity-backup/duplicity-backup.conf -b

Watch out for the schedule time of the automysqlbackup, you should schedule the duplicity backup job after the automysqlbackup job is done.

Finally backup your configurations.

cd ~
sudo /usr/local/src/duplicity-backup/duplicity-backup.sh -c /etc/duplicity-backup/duplicity-backup.conf --backup-script

Answert the prompt as showed bleow.

>> Are you sure you want to do that ('yes' to continue)?
yes

Enter passphrase: [gpg passphrase]

Source

Automatic Amazon s3 Backups on Ubuntu / Debian
GitHub duplicity-backup by zertrin
GitHub issue: No module named gio
Missing modules for paramiko and gio in duplicity foo

Backup Active Directory Group Policies with PowerShell

Tue, 22 Apr 2014 08:02:13 +0000

Based on my last Active Directory backup script I’ve developed a similar script to backup all group policies.

What it does:

Create a daily full snapshot of all group policy objects
Keep a daily, weekly and monthly snapshot
Notify me if something failed (requires PowerShell PowerUp)

<#
$Metadata = @{
    Title = "Backup Active Directory Group Policies"
    Filename = "Backup-ADGroupPolicies.ps1"
    Description = ""
    Tags = "backup, active, directory, group, object, policy"
    Project = ""
    Author = "Janik von Rotz"
    AuthorContact = "https://janikvonrotz.ch"
    CreateDate = "2014-04-22"
    LastEditDate = "2014-04-22
    Url = ""
    Version = "0.0.0"
    License = @'
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Switzerland License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ch/ or 
send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
'@
}
#>

try{

    #--------------------------------------------------#
    # modules
    #--------------------------------------------------#
    
    Import-Module ActiveDirectory
    Import-Module GroupPolicy
    
    #--------------------------------------------------#
    # settings
    #--------------------------------------------------#

    $Path = "C:\backup\GroupPolicy"

    #--------------------------------------------------#
    # main
    #--------------------------------------------------#

    # create backup file name
    $Filename = "GroupPolicyFull" + "#" + $((Get-Date -Format s) -replace ":","-") + ".bak"
    $Filepath = Join-Path $Path $Filename

    # backup active directory
    Get-GPO -All | ForEach-Object{
    
        $GPOFilepath = Join-Path $Filepath $_.DisplayName    
        New-Item -Path $GPOFilepath -ItemType Directory
        Backup-GPO -Guid $_.ID -Path $GPOFilepath    
    }
    
    # get dates for backup retention exclusion
    $Today = Get-Date -Format d
    $FirstDateOfWeek = Get-Date (Get-Date).AddDays(-[int](Get-Date).Dayofweek) -Format d
    $FirstDateOfMonth = Get-Date -Day 1 -Format d

    # delete all backups except for today, first day of week and first day of month
    Get-ChildItem $Path | select *,@{L="CreationTimeDate";E={Get-Date $_.CreationTime -Format d}} | Group-Object CreationTimeDate | %{
        
        # only one backup per day
        if($_.Count -gt 1){
            
            $_.Group | Sort-Object CreationTime -Descending | Select-Object -Skip 1     
        }
                
        # keep only required backups
        $_.Group | Where-Object{$_.CreationTimeDate -ne $Today -and $_.CreationTimeDate -ne $FirstDateOfWeek -and $_.CreationTimeDate -ne $FirstDateOfMonth}
            
    } | Remove-Item -Recurse -Force
    
}catch{

    Write-PPErrorEventLog -Source "Backup ActiveDirectory Group Policies" -ClearErrorVariable
}

Latest version of this script: https://gist.github.com/11167763

Backup Active Directory with PowerShell

Tue, 15 Apr 2014 13:57:06 +0000

Based on this Technet article I’ve developed a simple Active Directory backup tool with PowerShell.

What it does:

Create a full snapshot from Active Directory
Keep a daily, weekly and monthly snapshot
Notify me if something failed (requires PowerShell PowerUp)

<#
$Metadata = @{
	Title = "Backup ActiveDirecotry"
	Filename = "Backup-ActiveDirectory.ps1"
	Description = ""
	Tags = "backup, active, directory, ntsutil"
	Project = ""
	Author = "Janik von Rotz"
	AuthorContact = "https://janikvonrotz.ch"
	CreateDate = "2014-04-15"
	LastEditDate = "2014-04-15"
	Url = ""
	Version = "0.0.0"
	License = @'
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Switzerland License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ch/ or 
send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
'@
}
#>

try{

    <#
    #--------------------------------------------------#
    # about
    #--------------------------------------------------#
    
    The restore and backup process is described here: http://technet.microsoft.com/en-us/library/dd581644(WS.10).aspx
    
    #>
    
    #--------------------------------------------------#
    # settings
    #--------------------------------------------------#
    
    $Path = "C:\Powershell-PowerUp\backup\ActiveDirectory"
    
    #--------------------------------------------------#
    # main
    #--------------------------------------------------#
    
    # create backup file name
    $Filename = "ADBackupFull" + "#" + $((Get-Date -Format s) -replace ":","-") + ".bak"
    $Filepath = Join-Path $Path $Filename
    
    # backup active directory
    Invoke-Expression 'ntdsutil "activate instance ntds" ifm "create full $Filepath" quit quit'
    
    # get dates for backup retention exclusion
    $Today = Get-Date -Format d
    $FirstDateOfWeek = Get-Date (Get-Date).AddDays(-[int](Get-Date).Dayofweek) -Format d
    $FirstDateOfMonth = Get-Date -Day 1 -Format d
    
    # delete all backups except for today, first day of week and first day of month
    Get-ChildItem $Path | select *,@{L="CreationTimeDate";E={Get-Date $_.CreationTime -Format d}} | Group-Object CreationTimeDate | %{
        
        # only one backup per day
        if($_.Count -gt 1){
            
            $_.Group | Sort-Object CreationTime -Descending | Select-Object -Skip 1     
        }
                
        # keep only required backups
        $_.Group | Where-Object{$_.CreationTimeDate -ne $Today -and $_.CreationTimeDate -ne $FirstDateOfWeek -and $_.CreationTimeDate -ne $FirstDateOfMonth}
            
    } | Remove-Item -Recurse -Force

}catch{

    Write-PPErrorEventLog -Source "Backup ActiveDirectory" -ClearErrorVariable
}

Latest version of this script: https://gist.github.com/10729700

Install automysqlbackup

Tue, 08 Apr 2014 08:33:11 +0000

This post is part of my Your own Virtual Private Server hosting solution project.
Get the latest version of this article here: https://gist.github.com/9488132.

Introduction

AutoMySQLBackup with a basic configuration will create Daily, Weekly and Monthly backups of one or more of your MySQL databases from one or more of your MySQL servers.

Requirements

Installation

First create the backup folder.

sudo mkdir /var/backups/mysql

Install the latest version of automysqlbackup from the official website.

sudo mkdir /usr/local/src/automysqlbackup
cd /usr/local/src/automysqlbackup
sudo wget http://switch.dl.sourceforge.net/project/automysqlbackup/AutoMySQLBackup/AutoMySQLBackup%20VER%203.0/automysqlbackup-v3.0_rc6.tar.gz
sudo tar xvzf automysqlbackup-v3.0_rc6.tar.gz
sudo ./install.sh

Answer the install wizard as showed below.

global configuration diectory: default
directory for the executable: default

Add read permissios for all in the config folder.

cd /etc/automysqlbackup
sudo chmod a+r ./*

Update the automysqlbackup config file.

sudo vi /etc/automysqlbackup/automysqlbackup.conf

Set credentials and update the backup path.

# Username to access the MySQL server e.g. dbuser
CONFIG_mysql_dump_username='root'

# Password to access the MySQL server e.g. password
CONFIG_mysql_dump_password='[password]'

# Backup directory location e.g /backups
CONFIG_backup_dir='/var/backups/mysql'

Unlock the latest backup in order to make backups for third party programs available

# Store an additional copy of the latest backup to a standard
# location so it can be downloaded by third party scripts.
CONFIG_mysql_dump_latest='yes'

Let’s schedule the backup job by adding a daily cron script.

sudo vi /etc/cron.daily/automysqlbackup

With the following content.

#!/bin/sh

/usr/local/bin/automysqlbackup /etc/automysqlbackup/automysqlbackup.conf

chown root.root /var/backup/mysql* -R
sudo chmod -R a-x+X /var/backup/mysql

Enable execution of the script for the owner.

sudo chmod o+x /etc/cron.daily/automysqlbackup

You can run the backup script manually.

cd /etc/cron.daily
sudo ./automysqlbackup

Checkout the backup folder.

cd /var//backup/mysql

Source

How To Backup MySQL Databases on an Ubuntu VPS by Digital Ocean
How to back up MySQL databases using AutoMySQLBackup by A2 hosting automysqlbackup website

Install SharePoint 2013 Three-tier Farm - Deploy the SQL Server Backup Job

Fri, 07 Feb 2014 10:24:03 +0000

This post of is part of my Install SharePoint 2013 Three-tier Farm project.

A clean SQL Server maintenance plan should include three processes, backup, integrity check and index optimize.

You could simply set up a SQL Server maintenance plan within the SQL Server management studio. But be aware that there are better solutions, E.g. Ola Hallengren’s SQL Server Maintenance Solution is my absolutely favourite.

Ola Hallengren is a Microsoft MVP who has simplified the installation of SQL Server maintenance plans. There are many other MVP’s who even recommend his solutions over the SQL Server management solution:

Brent Ozar

However the installation of the SQL ServerMaintenance Solution is very easy. A little bit more difficult is the scheduling, that’s why I create a clone repository for Ola’s SQL scripts and added and example of how to schedule these jobs.

You’ll find the repository on GitHub: https://codeberg.org/janikvonrotz/SQL-Server-Maintenance

Office365 SharePoint Backup und Restore Facts by IOZ AG

Mon, 12 Aug 2013 13:33:30 +0000

In diesem Blogbeitrag schildern wir Ihnen die Fakten der Datensicherung Ihrer SharePoint Online Plattform:

SiteCollection

Papierkorb

via IOZ AG Blog.