Software-Configuration-Management on Janik von Rotz

Pin and update Ubuntu packages with Ansible

Fri, 05 Jun 2020 20:07:00 +0200

Patching is an essential part in server maintenance. For Windows servers and clients you have WSUS to coordinate updates and security patches. But how do you patch Ubuntu, Debian, RHEL or Fedora machines? The linux distro field is very dispersed, how do you do patch all these systems? In this post I will show you how I install package updates and ensure that specific packages cannot be updated using version pinning.

The current state of patch managenent.

Enterprise derivatives

RHEL and Ubuntu are both enterprise derivates, which means that enterprise features for these distros are managed by companies and not communities.

Ubuntu is managed by Canonical and they have their Livepatch Service to apply patches.

RHEL is managed by Red Hat and they provide Satellite to apply patches, system configurations and much more.

Community derivates

Debian, Fedora and many more are distros managed by communities. If you search their name in combination with patching you end up with various blog posts, where people show how they install the kernel updates and security patches. It is obvious that patch management is executed by companies running a farm of servers and there for seek automated solutions.

However, with Ansible you can easily manage you server farm and therefore also require an advanced solution to apply patches for your hosts.

Patch Ubuntu using Ansible

First let me assure that the method for patching can also be used for other distros. For better understanding I will show how its done using Ubuntu.

In this tutorial I will show you how I install essential packages and apply system patches. For my system configurations I try to be as explicit as possible, which means I pin versions for installed software. Assume that we have a Docker installation on an Ubuntu server and want to make sure that Docker is not updated as we do not want break compatibility.

Project setup

In my Ansible project I have created the following inventory and role:

inventories/setup
├── group_vars
│   ├── all.yml
│   └── ubuntu1804.yml
├── host_vars
│   ├── server1.yml
│   └── server2.yml
└── hosts.yml

roles/update/tasks
├── main.yml
└── update-ubuntu.yml

roles/docker/tasks
├── main.yml
└── install.yml

setup.yml

The setup.yml is an Ansible playbook that installs docker and updates servers. Note that I simplified the project setup for this tutorial.

Group vars

The group vars usually contains a list of packages.

inventories/setup/group_vars/ubuntu1804.yml

docker_package: "docker-ce=5:19.03.8~3-0~ubuntu-bionic"

I will demonstrate the pin version feature using this package.

Host vars

The host vars simply tell wether this host should be updated or not.

inventories/setup/host_vars/server1.yml

update: yes

Setup role

The setup role installs docker ce and pins its version if the package string contains an =.

roles/docker/tasks/install.yml

- name: Unpin docker-ce version
  dpkg_selections:
    name: "{{ docker_package }}"
    selection: hold
  when: "'=' not in docker_package"

- name: Update apt and install docker-ce
  apt:
    update_cache: yes
    name: "{{ docker_package }}"

- name: Pin docker-ce version
  dpkg_selections:
    name: "{{ docker_package.split('=')[0] }}"
    selection: hold
  when: "'=' in docker_package"

Once a package has been pinned, it cannot be updated by the package manger.

Update role

The update role runs only if updates are enabled for the host.

update/tasks/main.yml

- name: Include update tasks
  include_tasks: update-ubuntu.yml
  when: update

The update role refreshes the repo cache and runs a dist upgrade.

update/tasks/update-ubuntu.yml

- name: Update apt-get repo and cache
  apt:
    update_cache: yes
    force_apt_get: yes
    cache_valid_time: 3600

- name: Upgrade all packages
  apt:
    upgrade: dist

- name: Check if reboot is required
  stat: 
    path: /var/run/reboot-required
  register: reboot_required

- name: Reboot system if required
  reboot:
    msg: Rebooting to complete system upgrade
    reboot_timeout: 120
  when: reboot_required.stat.exists

If a reboot is required it will do so.

Run the playbook

The Ansible playbook setup.yml supports two running modes. The default mode and the update mode (update tag is required).

- hosts: all
  become: true
  roles:
  - role: docker
    tags: docker
  - role: update
    tags: update,never

To install docker on the target host you would enter: ansible-playbook -i inventory/setup setup.yml -l server1

And to update the target host you would enter: ansible-playbook -i inventory/setup setup.yml -l server1 -t update

It is recommend the run the update command periodically.

I hope I was able to demonstrate how you can manage patches for your linux systems using Ansible. As mentioned this method can be used for other non-aptitude-enabled distros.

Package Java Spring Boot service into rpm

Fri, 12 Apr 2019 11:38:15 +0200

This post is a follow-up to my last post The final rpm packaging guide. What I did not cover in the “final” guide was how to deal with the common case of packaging a service. In this post we are going to build a simple java spring boot application and package it as a systemd service into an rpm.

To get started I recommend to walk through the test setup section of my last post.

Requirements

To accomplish this tutorial a few dev tools must be installed.

httpie - Modern alternative to wget and curl.

sudo yum install httpie

rpmbuild - Utility to build rpms.

sudo yum install rpm-build

rpmdev - Dev utilities to build rpms.

sudo yum install rpmdevtools

rpmlint - Tool for checking common errors in rpm packages.

sudo yum install rpmlint

Example Application

We assume that we want to deploy a spring boot application.

Lets create one from the spring boot getting started source.

Checkout the official getting started repository.

cd ~ && git clone https://github.com/spring-guides/gs-spring-starter.git

Navigate into the project.

gs-spring-starter/initial

Then build and run the jar file.

./gradlew build && java -jar build/libs/gs-spring-starter-0.1.0.jar

Check if you get a response from the spring boot service.

http GET localhost:8080

It should return:

HTTP/1.1 200 
Content-Length: 27
Content-Type: text/plain;charset=UTF-8
Date: Fri, 12 Apr 2019 07:28:37 GMT

Greetings from Spring Boot!

Project Setup

Create the initial rpm folder structure.

cd ~; mkdir rpmbuild; cd ~/rpmbuild; rpmdev-setuptree

Copy the jar file into the source folder.

cp ../gs-spring-starter/initial/build/libs/gs-spring-starter-0.1.0.jar SOURCES/

Our example service will be managed by systemd and thus we also have to package a service file into the rpm.

Create the file vim SOURCES/spring-starter.service with the following content:

[Unit]
Description=Spring Starter
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/var/opt/spring-starter
ExecStart=/usr/bin/java -jar /usr/local/spring-starter/gs-spring-boot.jar
Restart=on-abort
User=spring-starter
Group=spring-starter

[Install]
WantedBy=multi-user.target

And thats it.

Write the Spec

The spec file defines the rpm build process and installation procedure.

Create a spec file for our spring starter application.

vim SPECS/spring-starter.spec

And set the following content:

##### HEADER SECTION #####

Name:           spring-starter
Version:        0.1.0
Release:        0
Summary:        Rpm package for Spring Starter

License:        ASL 2.0
URL:            https://spring.io
Source0:        gs-spring-boot-%{version}.jar
Source1:				%{name}.service

Requires:       shadow-utils,bash
BuildRequires:	systemd
%{?systemd_requires}

BuildArch:      noarch

%description
%{summary}

# disable debuginfo, which is useless on binary-only packages
%define debug_package %{nil}

# do not repack jar files
%define __jar_repack %{nil}

##### PREPARATION SECTION #####
%prep

# empty section

##### BUILD SECTION #####
%build

# empty section

##### PREINSTALL SECTION #####
%pre

# create Spring Starter service group
getent group spring-starter >/dev/null || groupadd -f -g 30000 -r spring-starter

# create Spring Starter service user
if ! getent passwd spring-starter >/dev/null ; then
    if ! getent passwd 30000 >/dev/null ; then
      useradd -r -u 30000 -g spring-starter -d /home/spring-starter -s /sbin/nologin -c "Spring Starter service account" spring-starter
    else
      useradd -r -g spring-starter -d /home/spring-starter -s /sbin/nologin -c "Spring Starter service account" spring-starter
    fi
fi
exit 0

##### INSTALL SECTION #####
%install

app_dir=%{buildroot}/usr/local/spring-starter
data_dir=%{buildroot}/var/opt/spring-starter
service_dir=%{buildroot}/%{_unitdir}

# cleanup build root
rm -rf %{buildroot}
mkdir -p  %{buildroot}

# create app folder
mkdir -p $app_dir

# create data folder
mkdir -p $data_dir

# create service folder
mkdir -p $service_dir

# copy all files
cp %{SOURCE0} $app_dir/gs-spring-boot.jar
cp %{SOURCE1} $service_dir

##### FILES SECTION #####
%files

# define default file attributes
%defattr(-,spring-starter,spring-starter,-)

# list of directories that are packaged
%dir /usr/local/spring-starter
%dir %attr(660, -, -) /var/opt/spring-starter

# list of files that are packaged
/usr/local/spring-starter/gs-spring-boot.jar
/usr/lib/systemd/system/%{name}.service

##### POST INSTALL SECTION #####
%post

# ensure Spring Starter service is enabled and running
%systemd_post %{name}.service
%{_bindir}/systemctl enable %{name}.service
%{_bindir}/systemctl start %{name}.service

##### UNINSTALL SECTION #####
%preun

# ensure Spring Starter service is disabled and stopped
%systemd_preun %{name}.service

%postun

case "$1" in
	0) # This is a package remove

		# remove app and data folders
		rm -rf /usr/local/spring-starter
		rm -rf /var/opt/spring-starter

		# remove Spring Starter service user and group
		userdel spring-starter
	;;
	1) # This is a package upgrade
		# do nothing
	;;
esac

# ensure Spring Starter service restartet if an upgrade is performed
%systemd_postun_with_restart %{name}.service

##### CHANGELOG SECTION #####
%changelog

* Wed Mar 20 2019 Janik vonRotz <contact@janikvonrotz.ch> - 0.1.0-0
- First spring-starter package

I wont go into details about the spec as I already did that in my last post. Not much has changed. However, one thing worth to mention is the %define __jar_repack %{nil} definition. This options disables the compression for .jar files. If the jar file is compressed it becomes inexecutable.

Further, as you might have noticed there are systemd macros, which ensure the service is configured properly. In the post install section two additional commands have been added to enable and start the service. This was done because t the systemd macros do not enable or start service by default.

Build

Build the rpm.

rpmbuild -ba SPECS/spring-starter.spec

Lint the rpm.

rpmlint RPMS/noarch/spring-starter-0.1.0-0.noarch.rpm

Rpmlint is very verbose, most warnings can be ignored.

Install the rpm.

sudo yum install RPMS/noarch/spring-starter-0.1.0-0.noarch.rpm -y

Check if the spring starter service is running.

systemctl status spring-starter

And have a look at the interface.

http GET localhost:8080

If the service responds you have accomplished the tutorial.

Uninstall the rpm.

sudo yum remove spring-starter -y

And thats it.

As usual feel free to ask any question in the comments.

See ya 😄

Helpers

Output rpm scriptlets.
rpm -E %{_unitdir}

Sources

Sources that I used to build this tutorial.

Fedora Packaging Guidelines - Systemd

Fedora Wiki - Packaging Systemd

Github - Systemd Macros Source

Golinuxhub - How to execute a script at %pre, %post, %preun or %postun stage (spec file) while installing/upgrading an rpm

The final rpm packaging guide

Wed, 20 Mar 2019 14:22:16 +0100

A customer required that every component of a software project I worked for is delivered as rpm.

This was problematic for Apache ActiveMQ Artemis as there is no rpm available.

But, the maintainer publishes a binary tarball and this is all we need to build an rpm.

So in this post I am going to show how to build an rpm from a tarball and explain the rpm build process in detail.

Introduction

I highly recommend to read the RPM Packaging Guide to get a basic understanding of the rpm build and packaging process.

Requirements

Lets get started by installing some dev tools.

httpie - Modern alternative to wget and curl.

sudo yum install httpie

rpmbuild - Utility to build rpms.

sudo yum install rpm-build

rpmdev - Dev utilities to build rpms.

sudo yum install rpmdevtools

rpmlint - Tool for checking common errors in rpm packages.

sudo yum install rpmlint

Test Setup

Before we setup the dev chain for the apache Artemis package, let’s get familiar with an hello world example.

Create a working dir wherever you would like.

mkdir rpmbuild

Create a symbolic to the rpmbuild folder from the home folder.

ln -s /path/to/rpmbuild ~/rpmbuild

Add default directories to build the RPM

cd ~/rpmbuild; rpmdev-setuptree

Macro Name	Name	Location	Purpose
`%_specdir`	Specification directory	`~/rpmbuild/SPECS`	RPM specifications (`.spec`) files
`%_sourcedir`	Source directory	`~/rpmbuild/SOURCES`	Pristine source package (e.g. tarballs) and patches
`%_builddir`	Build directory	`~/rpmbuild/BUILD`	Source files are unpacked and compiled in a subdirectory underneath this.
`%_buildrootdir`	Build root directory	`~/rpmbuild/BUILDROOT`	Files are installed under here during the `%install` stage.
`%_rpmdir`	Binary RPM directory	`~/rpmbuild/RPMS`	Binary rpms are created and stored under here.
`%_srcrpmdir`	Source RPM directory	`~/rpmbuild/SRPMS`	Source rpms are created and stored here.

Create spec file for the hello world example.

touch ~/rpmbuild/SPECS/hello-world.spec

And set the following content.

Name:       hello-world
Version:    1
Release:    0
Summary:    Most simple RPM package
License:    FIXME

%description
This is my first RPM package, which does nothing.

%prep
# we have no source, so nothing here

%build
cat > hello-world.sh <<EOF
#!/usr/bin/bash
echo Hello world
EOF

%install
mkdir -p %{buildroot}/usr/bin/
install -m 755 hello-world.sh %{buildroot}/usr/bin/hello-world.sh

%files
/usr/bin/hello-world.sh

%changelog
# let skip this for now

Build the hello world rpm.

cd ~/rpmbuild; rpmbuild -ba SPECS/hello-world.spec

Install the rpm.

sudo yum install RPMS/x86_64/hello-world-1-0.x86_64.rpm -y

Run hello world.

hello-world.sh

If the installation was successful you should see now the output: hello world.

Uninstall the hello world rpm.

sudo yum remove hello-world -y

Now the hello world command must not be available.

I assume that the hello world spec file is easy to understand. The install section might be misleading, but I will cover that in the next chapter.

Lets move with the more complex case.

Apache Artemis Spec File

Create the spec file for the apache Artemis.

touch ~/rpmbuild/SPECS/apache-artemis.spec

Navigate into the sources folder.

cd ~/rpmbuild/SOURCES

And download the apache Artemis binary.

http -d http://www.pirbot.com/mirrors/apache/activemq/activemq-artemis/2.6.4/apache-artemis-2.6.4-bin.tar.gz

Sections

The spec file has been split into multiple sections. Copy and paste the snippet for every section and read the notes carefully.

##### HEADER SECTION #####

Name:           apache-artemis
Version:        2.6.4
Release:        0
Summary:        Rpm package for Apache ActiveMQ Artemis

License:        ASL 2.0
URL:            http://activemq.apache.org/artemis/
Source0:        %{name}-%{version}-bin.tar.gz

Requires:       shadow-utils,bash

BuildArch:      noarch

%description
%{summary}

# disable debuginfo, which is useless on binary-only packages
%define debug_package %{nil}

In the header section metadata is defined for the build process and for the package itself.

Build

##### PREPARATION SECTION #####
%prep

# unpack tarball
%setup -q

##### BUILD SECTION #####
%build

# empty section

We use a tarball source, so no build step apart from unpacking the file is required. The %setup is a macro that extracts the tarball.

Install

##### PREINSTALL SECTION #####
%pre

# create Apache Artemis service group
getent group artemis >/dev/null || groupadd -f -g 30101 -r artemis

# create Apache Artemis service user
if ! getent passwd artemis >/dev/null ; then
    if ! getent passwd 30101 >/dev/null ; then
      useradd -r -u 30101 -g artemis -d /home/artemis -s /sbin/nologin -c "Apache Artemis service account" artemis
    else
      useradd -r -g artemis -d /home/artemis -s /sbin/nologin -c "Apache Artemis service account" artemis
    fi
fi
exit 0

##### INSTALL SECTION #####
%install

app_dir=%{buildroot}/opt/app/artemis
data_dir=%{buildroot}/opt/data/artemis

# cleanup build root
rm -rf %{buildroot}
mkdir -p  %{buildroot}

# create app folder
mkdir -p $app_dir

# create data folder
mkdir -p $data_dir

# copy all files
cp LICENSE $app_dir
cp README.html $app_dir
cp -R bin $app_dir
cp -R lib $app_dir
cp -R schema $app_dir
cp -R web $app_dir

Here we create the Artemis service user and group and copy selected files from the build to buildroot folder.

The %install section of an rpm spec file is not run on rpm package installation.

The %install section is run during package creation to install the files that need to be packaged such that the rpmbuild process can package them up.

File

##### FILES SECTION #####
%files

# define default file attributes
%defattr(-,artemis,artemis,-)

# list of directories that are packaged
%dir /opt/app/artemis
/opt/app/artemis/bin/*
/opt/app/artemis/lib/*
/opt/app/artemis/schema/*
/opt/app/artemis/web/*
%dir %attr(660, -, -) /opt/data/artemis

# list of files that are packaged
%doc /opt/app/artemis/README.html
%license /opt/app/artemis/LICENSE

The %files section lists all the files and directories that the package contains. The uninstallation process for an rpm is simply the removal of all the packaged files.

Uninstall

##### UNINSTALL SECTION #####
%postun

case "$1" in
	0) # This is a package remove

		# remove app and data folders
		rm -rf /opt/app/artemis
    	rm -rf /opt/data/artemis

		# remove Apache Artemis service user and group
  		userdel artemis
	;;
	1) # This is a package upgrade
  		# do nothing
	;;
esac

If additional work needs to be done before or after the files are removed the %preun and %postun scriptlets are available in the spec file for that work.

Changelog

##### CHANGELOG SECTION #####
%changelog

* Wed Mar 20 2019 Janik vonRotz <contact@janikvonrotz.ch> - 2.6.4-0
- Init apache-artemis package

Every time you make change, that is, whenever version or release of the package is incremented, add a comment to the changelog section

Full Spec File

The final spec file should look like this.

##### HEADER SECTION #####

Name:           apache-artemis
Version:        2.6.4
Release:        0
Summary:        Rpm package for Apache ActiveMQ Artemis

License:        ASL 2.0
URL:            http://activemq.apache.org/artemis/
Source0:        %{name}-%{version}-bin.tar.gz

Requires:       shadow-utils,bash

BuildArch:      x86_64

%description
%{summary}

# disable debuginfo, which is useless on binary-only packages
%define debug_package %{nil}

##### PREPARATION SECTION #####
%prep

# unpack tarball
%setup -q

##### BUILD SECTION #####
%build

# empty section

##### PREINSTALL SECTION #####
%pre

# create Apache Artemis service group
getent group artemis >/dev/null || groupadd -f -g 30101 -r artemis

# create Apache Artemis service user
if ! getent passwd artemis >/dev/null ; then
    if ! getent passwd 30101 >/dev/null ; then
      useradd -r -u 30101 -g artemis -d /home/artemis -s /sbin/nologin -c "Apache Artemis service account" artemis
    else
      useradd -r -g artemis -d /home/artemis -s /sbin/nologin -c "Apache Artemis service account" artemis
    fi
fi
exit 0

##### INSTALL SECTION #####
%install

app_dir=%{buildroot}/opt/app/artemis
data_dir=%{buildroot}/opt/data/artemis

# cleanup build root
rm -rf %{buildroot}
mkdir -p  %{buildroot}

# create app folder
mkdir -p $app_dir

# create data folder
mkdir -p $data_dir

# copy all files
cp LICENSE $app_dir
cp README.html $app_dir
cp -R bin $app_dir
cp -R lib $app_dir
cp -R schema $app_dir
cp -R web $app_dir

##### FILES SECTION #####
%files

# define default file attributes
%defattr(-,artemis,artemis,-)

# list of directories that are packaged
%dir /opt/app/artemis
/opt/app/artemis/bin/*
/opt/app/artemis/lib/*
/opt/app/artemis/schema/*
/opt/app/artemis/web/*
%dir %attr(660, -, -) /opt/data/artemis

# list of files that are packaged
%doc /opt/app/artemis/README.html
%license /opt/app/artemis/LICENSE

##### UNINSTALL SECTION #####
%postun

case "$1" in
	0) # This is a package remove

		# remove app and data folders
		rm -rf /opt/app/artemis
    	rm -rf /opt/data/artemis

		# remove Apache Artemis service user and group
  		userdel artemis
	;;
	1) # This is a package upgrade
  		# do nothing
	;;
esac

##### CHANGELOG SECTION #####
%changelog

* Wed Mar 20 2019 Janik vonRotz <contact@janikvonrotz.ch> - 2.6.4-0
- Init apache-artemis package

Installation

Navigate to the rpm build directory

cd ~/rpmbuild

Build the rpm.

rpmbuild -ba SPECS/apache-artemis.spec

Lint the rpm.

rpmlint RPMS/x86_64/apache-artemis-2.6.4-0.x86_64.rpm

There will be thousands of error and warnings. Ignore them for now.

Install the rpm.

sudo yum install RPMS/x86_64/apache-artemis-2.6.4-0.May.rpm -y

Create a broker instance.

sudo /opt/app/artemis/bin/artemis create broker /opt/data/artemis/broker --user artemis --password artemis --require-login

Start the Artemis instance.

sudo "/opt/data/artemis/broker/bin/artemis-service" start

Open the browser to localhost:8161/console and check if you can login with artemis:artemis.

If everything worked you can stop the service.

sudo "/opt/data/artemis/broker/bin/artemis-service" stop

And check if there is really no Artemis service running.

ps aux | grep artemis

Finally remove the package.

sudo yum remove apache-artemis -y

Disclaimer

Apache Artemis has been installed to /opt/app/artemis and the instance has been created in /opt/data/artemis. Note that this is not best practice. The Filesystem Hierarchy Standard tells where to put what.

Conclusion

That’s it! Lets recap what we just learned.

First we created the rpmbuild folder structure
Then created an example spec file for an hello world script
Next we prepared the assets for the Apache Artemis package
We defined the spec file for the rpm
Following-up we installed the rpm
Then created, started, tested and killed an Artemis instance
Finally we uninstalled the Apache Artemis package

This is also resembles the life cycle of an rpm service.

As usual if you have any questions, concerns or inputs feel free to make a comment.

Edits

Edit 1: Upgrade

I also investigated the behavior in an upgrade scenario.

Assuming the following points apply:

Release or version is incremented
A file e.g. README.html has been removed
An rpm is already installed
Yum does an rpm install

The results are as followed:

Yum detects the upgrade
The README file is removed

It behaves as expected.

Edit 2: Gradle Project

I have created a gradle project to facilitate the build process.

Link: GitHub - Apache Artemis RPM

Sources

Sources I relied on to build this tutorials:

Fedora Packaging Guidelines

RPM Packaging Guide

superuser - Managing service accounts in an RPM spec

packagecloud - Building RPM packages with rpmbuild

Generate PEM key- and truststores With Puppet

Thu, 07 Mar 2019 10:23:04 +0100

This post is a follow-up of Generate pkcs12 key- and truststores with Puppet.
In this post we are going to create PEM key- and truststores with Puppet.

PEM files are base64 encoded X.509 certificates. Enclosed between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- multiple PEM files can be concatinated into key- and truststores. And that is exactly what we are going to do using a Puppet manifest.

For the example use case multiple certificate files are required. Using the commands from the Create a certificate authority and sign server certificates without prompting using openssl post the following files must be provided:

localhost_cert.pem
localhost_key.pem
example.com_cert.pem
ca_cert.pem

The certificates will be concatinated into the following stores:

webservice-keystore.pem
- localhost - private key entry
webservice-truststore.pem
- ca - trusted cert entry
custom-truststore.pem
- example.com - trusted cert entry
- ca - trusted cert entry

All the files are processed in /var/tmp/certificates.

The manifest file is defined as followed:

modules/certbox/manifests/init.pp

class certbox (

    String $host = 'localhost',
    String $cert_dir = '/var/tmp/certificates',

    String $server_ca_cert = "${cert_dir}/ca_cert.pem",
    String $server_cn = $host,
    String $server_cert = "${cert_dir}/${host}_cert.pem",
    String $server_key = "${cert_dir}/${host}_key.pem",
    String $server_key_pass = 'password',
    String $server_keystore = "${cert_dir}/webservice-keystore.pem",
    String $server_truststore = "${cert_dir}/webservice-truststore.pem",

    Array $custom_trust_entries = [
      {
        cn => 'example.com',
        cert => 'example.com_cert.pem'
      },
      {
        cn => 'ca',
        cert => 'ca_cert.pem'
      },
    ],
    String $custom_truststore = "${cert_dir}/custom-truststore.pem",

    String $owner = 'root',
    String $group = 'root',
    String $file_read_mode = 'a=,ug+r',

) {

  # component-wide defaulting of the exec path attribute
  Exec {
    path => ['/usr/bin', '/bin']
  }

  # create copy of sever cert, if file state changes notify keystore assemble task
  file { "${server_cert}.tmp":
    ensure => file,
    owner  => $owner,
    group  => $group,
    mode   => $file_read_mode,
    source => $server_cert,
    notify => Exec["certbox - create server keystore"],
  }

  # check if keystore file does not exist and notify creation task
  exec { "certbox - check if server keystore exists":
    onlyif => "test ! -f $server_keystore",
    command => 'echo "File does not exist."',
    notify => Exec["certbox - create server keystore"],
  }

  # create server keystore if temporary file state changed
  exec { "certbox - create server keystore":
    refreshonly => true,
    command     => "cat ${server_cert} > ${server_keystore}; cat ${server_key} >> ${server_keystore}",
  }

  # ensure server keystore file permissions
  file { $server_keystore:
    ensure => file,
    owner  => $owner,
    group  => $group,
    mode   => $file_read_mode,
  }

  file { "${server_ca_cert}.tmp":
    ensure => file,
    owner  => $owner,
    group  => $group,
    mode   => $file_read_mode,
    source => $server_ca_cert,
    notify => Exec["certbox - create server truststore"],
  }

  exec { "certbox - check if server truststore exists":
    onlyif => "test ! -f $server_truststore",
    command => 'echo "File does not exist."',
    notify => Exec["certbox - create server truststore"],
  }

  exec { "certbox - create server truststore":
    refreshonly => true,
    command     => "cat ${server_ca_cert} > ${server_truststore}",
  }

  file { $server_truststore:
    ensure => file,
    owner  => $owner,
    group  => $group,
    mode   => $file_read_mode,
    notify => Exec["certbox - create server truststore"],
  }

  # create copy for each ca truststore entry
  $custom_trust_entries.each | Integer $index, Hash $entry | {

    file { "${cert_dir}/${entry['cert']}.ca_trust.icam.tmp":
      ensure => file,
      owner  => $owner,
      group  => $group,
      mode   => $file_read_mode,
      source => "${cert_dir}/${entry['cert']}",
      notify => [
        Exec["certbox - cleanup ca truststore"],
        Exec["certbox - add pem ${entry['cn']} to ca truststore"]],
    }

    exec { "certbox - check if ca truststore for ${entry['cn']} exists":
      onlyif => "test ! -f $custom_truststore",
      command => 'echo "File does not exist."',
      notify => Exec["certbox - add pem ${entry['cn']} to ca truststore"],
    }
  }

  # reset ca truststore if temporary entry file state changed
  exec { "certbox - cleanup ca truststore":
    refreshonly => true,
    command     => "echo > ${custom_truststore}",
  }

  # add ca truststore entry if temporary entry file state changed
  $custom_trust_entries.each | Integer $index, Hash $entry | {

    exec { "certbox - add pem ${entry['cn']} to ca truststore":
      refreshonly => true,
      command     => "echo >> ${custom_truststore};cat ${cert_dir}/${entry['cert']} >> ${custom_truststore}",
    }
  }

  file { $custom_truststore:
    ensure  => file,
    owner   => $owner,
    group   => $group,
    mode    => $file_read_mode,
  }
}

This manifest checks if the source PEM files have changed or the target stores does not exist. If one of these cases apply the manifests creates the key- or truststore. Finally, it sets the file mode and ownership.

To apply the manifest with Puppet run the following command:

sudo puppet apply --modulepath=modules/ -e "include certbox"

To verify if the files have been assembled correctly use this keytool commands:

sudo keytool -printcert -file /var/tmp/certificates/custom-truststore.pem

Let me know if this post helped you to resolve a particular use case.

Generate pkcs12 key- and truststores with Puppet

Wed, 30 Jan 2019 11:16:52 +0100

In my last post I have showed how to generate pkcs12 key- and truststores using openssl and keytool.

For this post we assume that we want to automate the store assembling with Puppet. Puppet is a configuration management tool that shares many ideas with Ansible. In the world of Puppet you define a manifest file that describes a state of how a file, service or any type of resource should look like. Puppet applies these manifests and makes sure that the targeted system reaches the defined state.

Using Puppets exec resource we are going to define key- and truststores using openssl and keytool.

Our manifest assure that the stores are assembled as defined. Puppet will be able to detect deltas and act accordingly.

For our use-case we assume we have generate multiple certificates under /var/tmp/certificates:

localhost_cert.pem
localhost_key.pem
example.com_cert.pem
example.com_key.pem
ca_cert.pem

Our fictive webservice requires a key- and truststore containing the provided certificates and keys:

webservice-keystore.pkcs12
- ca - trusted cert entry
- localhost - private key entry
- example.com - private key entry
webservice-truststore.pkcs12
- ca - trusted cert entry

Puppet is installed on our system and we are ready to declare and apply our manifest file.

Our Puppet module will be called certbox. Modules must follow a strict naming and folder structure.

Below is a copy-and-paste definition for our Puppet module. Create the manifest file and folders as showed.

modules/certbox/manifests/init.pp

class certbox (

  String $certDir = "/var/tmp/certificates",

  String $caCert = "$certDir/ca_cert.pem",

  String $cn1 = "localhost",
  String $cert1 = "$certDir/${cn1}_cert.pem",
  String $key1 = "$certDir/${cn1}_key.pem",
  String $keyPassword1 = "password",

  String $cn2 = "example.com",
  String $tmpKeystore2 = "$certDir/$cn2.pkcs12",
  String $cert2 = "$certDir/${cn2}_cert.pem",
  String $key2 = "$certDir/${cn2}_key.pem",
  String $keyPassword2 = "password",

  String $keystore = "$certDir/webservice-keystore.pkcs12",
  String $truststore = "$certDir/webservice-truststore.pkcs12",
  String $keystorePassword = "password",
  String $truststorePassword = "password",

  String $owner = "root",
  String $group = "root",
  String $fileReadMode = "a=,ug+r",

) {

  exec { "remove keystore for $cn1 if password changed or is empty":
    onlyif => "/bin/keytool -list -storetype PKCS12 -keystore $keystore -storepass $keystorePass | grep 'password was incorrect\\|file exists, but is empty'",
    command => "/bin/rm $keystore1",
  }

  exec { "create pkcs12 keystore for $cn1":
    onlyif => "/bin/keytool -list -keystore $keystore -storepass $keystorePassword | grep $(openssl x509 -noout -fingerprint -sha1 -in $cert1 | cut -f2 -d \"=\");test $? -eq 1",
    command => "/bin/openssl pkcs12  -export -in $cert1 -inkey $key1 -passin pass:$keyPassword1 -certfile $caCert -out $keystore -passout pass:$keystorePassword -name $cn1",
  }

  exec { "remove keystore for $cn2 if password changed or is empty":
    onlyif => "/bin/keytool -list -storetype PKCS12 -keystore $tmpKeystore2 -storepass $keystorePass | grep 'password was incorrect\\|file exists, but is empty'",
    command => "/bin/rm $tmpKeystore2",
  }

  exec { "create pkcs12 keystore for $cn2":
    onlyif => "/bin/keytool -list -keystore $tmpKeystore2 -storepass $keystorePassword | grep $(openssl x509 -noout -fingerprint -sha1 -in $cert2 | cut -f2 -d \"=\");test $? -eq 1",
    command => "/bin/openssl pkcs12 -in $cert2 -inkey $key2 -passin pass:$keyPassword2 -export -out $tmpKeystore2 -passout pass:$keystorePassword -name $cn2",
  }

  exec { "merge $cn2 into $cn1 keystore":
    onlyif => "/bin/keytool -list -keystore $keystore -storepass $keystorePassword | grep $(openssl x509 -noout -fingerprint -sha1 -in $cert2 | cut -f2 -d \"=\");test $? -eq 1",
    command => "/bin/keytool -importkeystore -storetype PKCS12 -destkeystore $keystore -deststorepass $keystorePassword -destkeypass $keystorePassword \
      -srckeystore $tmpKeystore2 -srcstoretype PKCS12 -srcstorepass $keystorePassword -alias $cn2 -noprompt",
  }

  file { $keystore:
    ensure => file,
    owner => $owner,
    group => $group,
    mode => $fileReadMode,
  }

  exec { "remove truststore for $cn1 if password changed":
    onlyif => "/bin/keytool -list -storetype PKCS12 -keystore $truststore -storepass $truststorePass | grep 'password was incorrect'",
    command => "/bin/rm $truststore",
  }

  exec { "create pkcs12 truststore":
    onlyif => "/bin/keytool -list -keystore $truststore -storepass $truststorePassword | grep $(openssl x509 -noout -fingerprint -sha1 -in $caCert | cut -f2 -d \"=\");test $? -eq 1",
    command => "/bin/keytool -importcert -storetype PKCS12 -keystore $truststore -storepass $truststorePassword -alias ca -file $caCert -noprompt",
  }

  file { $truststore:
    ensure => file,
    owner => $owner,
    group => $group,
    mode => $fileReadMode,
  }
}

Edit 1: Compare sha1 of cert to assert if import should be executed.
Edit 2: On create keystore check not if store file already exist, but check if cert in keystore matches the sha1. The check should also act as expected if file does not exist.
Edit 3: Remove the create empty truststore task. If the truststore file does not exist, the keytool import command will create the file.
Edit 4: Added exec task that remove the store file if the password has changed.

The manifest is kept fairly simple. However, you might ask why we have to merge our second certificate from a pkcs12 store. The answer is a bit more complicated. The openssl and keytool utilities help generating and managing key material. They provide similar tasks, but differ heavy in specific features. Here are the most important differences:

Openssl cannot manipulate existing pkcs12 stores.
Openssl cannot create pkcs12 with multiple certificates and keys.
Keytool cannot import certificates and keys into an existing pkcs12 store, however, it can import a pkcs12 store into an existing one.
Keytool cannot create pkcs12 stores from a certificate and key.

Hope this helps to understand the design decisions made here much better.

To apply the manifest with Puppet run the following command:

sudo puppet apply --modulepath=modules/ -e "include certbox"

At this point Puppet should create the key- and truststores.

If you ended up with an error or faced some other issues let me know!

Puppet masterless project setup guide

Sat, 08 Dec 2018 09:25:50 +0100

This document is a proposal for a Puppet project setup. It covers the setup of masterless Puppet module and provides a layout for a proper project structure. It will give you an example for a nginx module that can be deployed locally and remote. Further the roles and profiles concept will be applied and coupled with the hiera configuration data.

Requirements

Walking through this guide requires specific abilities and tools:

Basic Puppet knowledge
Working in a unix environment
Access to a git repository

Also prepare the following values:

_PROJECT_NAME_: Name of the git repo
_GIT_REPO_URL_: Git repo url for pushing the project

And export them:

export _PROJECT_NAME_="value"
export _GIT_REPO_URL_="value"

Project Setup

Install Puppet and Git on your development machine.

yum install puppet git -y

Make sure at least Puppet version 5 has been installed

Optional:

Create a git repository and push an initial commit.

mkdir ~/$_PROJECT_NAME_
git init
touch README.md
git add .
git commit -m "init"
git remote add origin $_GIT_REPO_URL_
git push -u origin master

Initial Puppet Module

Next we are going to create a folder and file structure for a dummy module.

cd ~/$_PROJECT_NAME_
mkdir -p ./modules/foo/files/
echo "hello world" > ./modules/foo/files/hello
mkdir -p ./modules/foo/manifests/
touch ./modules/foo/manifests/init.pp
mkdir ./manifests/
echo "include foo" > ./manifests/site.pp

Add the following definitions to the init.pp file

init.pp

class foo {
  file { "/tmp/hello":
    ensure => file,
    source => "puppet:///modules/foo/hello",
  }
}

We added a Puppet module named foo that creates a file and is included in our site.pp.

The Puppet code can be executed with the following command:

puppet apply --modulepath modules manifests/site.pp

You should find a file /tmp/hello with content hello world.

If everything worked, we are ready to extend the project structure.

Hiera Files

Based on facts (Puppet environment variables) we assign to a node (host) our Puppet setup should load specific configurations and apply them to modules. The Puppet hiera concept helps us achieving that.

First you need to now about facts.

Facts are key values that are available to Puppet by default.

Show all available facts by running facter.

Create a custom fact: export FACTER_environment="dev"

Show the custom fact: facter environment

Simple isn’t it?

Depending on these facts and the Puppet configuration our project should load specific configuration data. The configuration data structure is organized as a hierarchy.

Our configuration hierarchy will consist of three levels:

common: Data loaded by default.
environment: Data assigned to an environment.
node: Data assigned to a host.

Create the folder structure and files for the hierarchy.

cd ~/$_PROJECT_NAME_
mkdir -p ./data/$(facter environment)/nodes/
touch ./data/common.yaml
touch ./data/$(facter environment)/default.yaml
touch ./data/$(facter environment)/nodes/$(facter hostname).yaml
touch hiera.yaml

Update the hiera configuration file.

hiera.yaml

---
version: 5

defaults:
  data_hash: yaml_data
  datadir: data

hierarchy:
  - name: "Per-node data"
    path: "%{facts.environment}/nodes/%{facts.hostname}.yaml"

  - name: "Per-environment defaults"
    path: "%{facts.environment}/default.yaml"

  - name: "Common data"
    path: "common.yaml"

The hiera file implements the mentioned data hierarchy.

Now we are going to set configurations for our data hierarchy.

data/common.yaml

nginx::port: 80

data/dev/default.yaml

nginx::host: default_server

data/dev/nodes/_HOSTNAME_.yaml

nginx::welcome_message: "hello world"

This configuration data will be applied to the nginx module.

Nginx Module

Create the nginx module folders and files.

cd ~/$_PROJECT_NAME_
mkdir -p ./modules/nginx/manifests/
touch ./modules/nginx/manifests/init.pp
mkdir -p ./modules/nginx/templates/
touch ./modules/nginx/templates/index.html.epp
touch ./modules/nginx/templates/nginx.conf.epp
echo "include nginx" > ./manifests/site.pp

Add resources to the nginx module.

init.pp (nginx)

class nginx (
  Integer $port,
  String $host,
  String $welcome_message,
) {

  package { "nginx":
    ensure => installed,
  }

  service { "nginx":
    ensure => running,
  }

  file { "/usr/share/nginx/html/index.html":
    ensure  => present,
    content => epp("nginx/index.html.epp"),
  }

  file { "/etc/nginx/nginx.conf":
    ensure  => present,
    content => epp("nginx/nginx.conf.epp"),
    notify => Service['nginx'],
  }
}

Update the index.html template.

index.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
  <head>
    <title>Test Page for the Nginx HTTP Server</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  </head>

  <body>
    <h1>Welcome to <strong>nginx</strong></h1>
    <p><%= $nginx::welcome_message %></p>
  </body>
</html>

Update the nginx conf template.

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

    server {
        listen       <%= $nginx::port %> <%= $nginx::host %>;
        listen       [::]:<%= $nginx::port %> <%= $nginx::host %>;
        server_name  _;
        root         /usr/share/nginx/html;

        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

And finally install everything.

puppet apply --modulepath modules manifests/site.pp --debug --hiera_config hiera.yaml

Open your browser to http://localhost:80/.

Roles and Profiles

Having multiple modules, shared resources and dependencies comes at cost of complexity. Thus Puppet came up with the roles and profiles concept.

The official documentation explains the key idea very well:

Roles and profiles are two extra layers of indirection between your node classifier and your component modules. This separates your code into three levels:

Component modules: Normal modules that manage one particular technology. (For example, puppetlabs/apache.)

Profiles: Wrapper classes that use multiple component modules to configure a layered technology stack.

Roles: Wrapper classes that use multiple profiles to build a complete system configuration.

And here is a visual presentation:

![puppet roles and profiles concept](/images/puppet roles and profiles concept.png)

Further roles and profiles allow us to decouple Puppet modules from the hiera config.

Create the profile and role files.

cd ~/$_PROJECT_NAME_
mkdir -p ./modules/profile/manifests
mkdir -p ./modules/role/manifests
touch ./modules/profile/manifests/nginx.pp
touch ./modules/role/manifests/webserver.pp

Further to make things a bit more interesting create a new module.

mkdir -p ./modules/base/manifests
touch ./modules/base/manifests/init.pp

init.pp (base)

class base (
  String $username,
) {
  user { $username:
    ensure => present,
  }
}

The profile contains multiple modules. Update the nginx profile as followed:

nginx.pp

class profile::nginx (
  Integer $port = '80',
  String $host = 'default_server',
  String $welcome_message = 'hello world',
) {

  class { "base":
    username => "dummy",
  }

  class { "nginx":
    port => $port,
    host => $host,
    welcome_message => $welcome_message,
  }
}

The profile helps to decouple modules from the hiera config.

A role includes multiple profiles and modules.

webserver.pp

class role::webserver {
  include foo
  include profile::nginx
}

In roles only include modules that do not need any configuration.

Update the hiera data with profile configuration keys.

data/common.yaml

profile::nginx::port: 81

data/dev/default.yaml

profile::nginx::host: default_server

data/dev/nodes/_HOSTNAME_.yaml

profile::nginx::welcome_message: "how are you?"

The hiera data will be looked up by the profile and applied to the nginx module.

As a final step assign the role to the default node.

site.pp

node default {
  include role::webserver
}

Run the new setup.

puppet apply --modulepath modules manifests/site.pp --debug --hiera_config hiera.yaml

Open your browser to http://localhost:81/.

And we are done 😄 Now you have masterless Puppet project up and running.

I hope you enjoyed this tutorial, if not let me know. Feel free to ask questions.

Source

Puppefy IT Automation - How To Set Up a Masterless Puppet on CentOS

GitHub Gist - Simplest example of a masterless puppet invocation using module paths

Puppet Ask - Masterless puppet with hiera dynamic lookup?

GitHub - Puppet Simple Masterless

ScaleScale - How to install Nginx on CentOS 6 / RHEL

Software-Configuration-Management on Janik von Rotz

Pin and update Ubuntu packages with Ansible

Enterprise derivatives

Community derivates

Patch Ubuntu using Ansible

Project setup

Group vars

Host vars

Setup role

Update role

Run the playbook

Package Java Spring Boot service into rpm

Requirements

Example Application

Project Setup

Write the Spec

Build

Helpers

Sources

The final rpm packaging guide

Introduction

Requirements

Test Setup

Apache Artemis Spec File

Sections

Header

Build

Install

File

Uninstall

Changelog

Full Spec File

Installation

Disclaimer

Conclusion

Edits

Edit 1: Upgrade

Edit 2: Gradle Project

Sources

Generate PEM key- and truststores With Puppet

Generate pkcs12 key- and truststores with Puppet

Puppet masterless project setup guide

Requirements

Project Setup

Initial Puppet Module

Hiera Files

Nginx Module

Roles and Profiles

Source