Driving OpenStack via Ansible

Last week I spoke at the Atlanta OpenStack meetup about “Driving OpenStack via Ansible,” in which I introduced Ansible as a tool and talked about its ability to integrate with OpenStack. As part of the presentation I used two playbooks to launch VMs on a cloud and configure them with different applications. We walked through the playbooks and talked about what they were doing, the things that tripped me up while writing them, and then brainstormed ways to use Ansible in situations that have come up for members of the meetup.

One playbook uses my role to install ZNC, the popular IRC “bouncer,” for maintaining a persistent chat presence. The other demo was based on a playbook with the roles needed to configure a server for OpenStack development, ready to run devstack.

The slides are available, and you can download the playbooks from the github repository and try them yourself.

We used Dreamhost’s public cloud, DreamCompute, for the demo at the meetup. Thanks to the DreamHost crew for providing those resources!

Continue reading “Driving OpenStack via Ansible”

Deploying Nested ZNC Services with Ansible

The OpenStack community, like other open source communities, relies on
IRC for a lot of our brief and interactive communication. The
community is large, though, and spans the globe, so we are not all
online simultaneously. Many of us also travel, meaning we are offline
at times even when we are otherwise “working.” The result is lags in
communication, or missed messages entirely.

ZNC is an IRC “bouncer”, a tool for maintaining a presence on an
Internet Relay Chat network even when you yourself are not connected
to the Internet. ZNC maintains a scroll-back buffer for each channel
you join, letting you replay missed messages after reconnecting to the
service for a while. Having access to the messages you’ve missed makes
IRC more useful as an asynchronous communication tool.

Advanced ZNC Configuration

Using a single ZNC server is a good solution if you use just one
computer to connect to IRC. For anyone with both a desktop and laptop,
or even a phone or tablet, though, using a single shared ZNC server
can actually result in losing messages. If your desktop computer is
on, it consumes the scroll-back so you can’t see them when your laptop
comes online, for example.

ZNC supports multiple users, so it is possible to configure it so that
each device has its own buffers. However, that means each user needs
to be connected to each channel in order to have a buffer
saved. Managing that by hand, by subscribing each user, is a pain.

Last year Sean Dague wrote a blog post describing a puppet module
he and Dan Smith created for managing multiple ZNC servers. The idea
they came up with is actually to nest the servers, so that one base
server talks to the IRC network(s) and the others serve the
scroll-back for each client (the diagram in the blog post explains the
configuration very well). They dubbed this configuration “ZNC on ZNC”.

znc-on-znc Ansible Role

I use Ansible, instead of puppet, for managing my development server
configurations. As a way to expand my Ansible knowledge, I decided to
build an Ansible role to implement the ZNC-on-ZNC configuration that
Sean and Dan had started. I ended up going a little further and adding
features to make it possible to connect the ZNC services to more than
one upstream network – so you can connect to a Bitlbee instance that
proxies to a Jabber network, for example.

The role can be installed from the git repository or with
ansible-galaxy:

$ ansible-galaxy install dhellmann.znc-on-znc

The README file describes all of the options in detail, and gives an
example configuration. The most important values are the znc_user
settings for securing the ZNC servers, the znc_networks for
managing the IRC networks to which the server connects upstream, and
the znc_clients list for defining how many and which clients will
be connecting to the servers.

What You Get

Each entry in znc_networks results in an upstream connection, and
each entry in znc_clients results in a separate ZNC process to
which a client can connect as the znc_user to access the private
scroll-back buffers.

Each ZNC instance is configured with a self-signed SSL certificate,
generated as part of the deployment. The base instance is configured to
connect to all of the networks with a set of channels to join, so if
the ZNC server loses the connection is can rejoin all of your channels
automatically.

The child instances are configured to connect to the server and trust
its SSL certificate. The child connects to the base server separately
for each network, and the subscribed channels are configured so that
all child instances have buffers for all channels.

The amount of buffering in each child is configurable separately, so
that small mobile devices don’t have an excessive amount of playback
every time they connect.

monitd is configured to keep all of the ZNC instance running, in
case any die or the server is rebooted.

Another useful feature is triggered by the
znc_firewall_bypass_port setting, which connects port 443 to one
of the selected clients to allow you to use your ZNC instance through
a firewall that blocks traffic on the normal IRC ports. The clients
don’t run as root, so they can’t bind directly to the privileged
port. Instead, rinetd connects port 443 with the unprivileged port
the client is configured to use normally.

Hosting ZNC

I use a small cloud instance to host the ZNC service, so it is
accessible from anywhere I might travel. If you don’t have a cloud
account, or don’t want to pay for a server to run ZNC, any machine
that maintains an Internet connection and to which you can connect
remotely will work as well.

Ansible Role for OpenStack Development

As mentioned previously, I have been using
ansible for managing my development configuration for a few months
now, and I’m finally getting around to releasing some of the roles
I’ve created in a form (possibly) useful to other folks. This one
configures some of the packages needed to run unit tests for OpenStack
projects.

Features

There’s nothing quite as annoying as having tox fail to build a test
environment because some system package is missing. The
openstack-dev role installs many of the commonly needed system
dependencies, such as libxml2 and libffi-dev. It isn’t an
exhaustive list, and it isn’t sufficient to run OpenStack properly,
but it does allow most of the unit tests to run, including some that
are optional when driver libraries are not present. For a complete
list of packages installed, refer to the source.

The role also sets up a working directory
(openstack_dev_repos_dir) where the git repositories containing
OpenStack source can be checked out. The role clones the
openstack/oslo-incubator and creates an update.sh script in
the openstack_dev_repos_dir to call the tool from the
oslo-incubator for cloning copies of all of the OpenStack source
trees.

The role also clones openstack/requirements, which contains
tools/build_wheels.sh, a script for building wheels of all of the
OpenStack project Python requirements. Since the role depends on
dhellmann.devpi, a devpi server is set up running on the
development server. Running build_wheels.sh will populate the
cache for the devpi server and the wheelhouse so that tox environment
builds for the unit tests for projects with a lot of dependencies will
go more quickly.

Installing the Role

Clone the role from github or use ansible-galaxy to install
dhellmann.openstack-dev. If you use ansible-galaxy, the
dependencies will also be installed automatically.

$ ansible-galaxy install dhellmann.openstack-dev
- downloading role 'openstack-dev', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-openstack-dev/archive/1.0.0.tar.gz
- extracting dhellmann.openstack-dev to roles/dhellmann.openstack-dev
- dhellmann.openstack-dev was installed successfully
- adding dependency: dhellmann.devpi
- adding dependency: dhellmann.python-dev
- downloading role 'devpi', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-devpi/archive/1.0.0.tar.gz
- extracting dhellmann.devpi to roles/dhellmann.devpi
- dhellmann.devpi was installed successfully
- downloading role 'python-dev', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-python-dev/archive/1.0.0.tar.gz
- extracting dhellmann.python-dev to roles/dhellmann.python-dev
- dhellmann.python-dev was installed successfully

Add the role to your playbook for your development server, and
optionally set openstack_dev_repos_dir.

Cloning the Source

After running the playbook, login to the server and:

$ cd ~/repos
$ ./update.sh

If you changed openstack_dev_repos_dir, replace the cd command
with the appropriate value.

Building Wheel Cache

$ cd ~/repos/openstack/requirements
$ ./tools/build_wheels.sh 2.7 3.4

The command will take some time, so you may want to get lunch while
you wait.

Ansible Roles for Python Developers

I have been using ansible for managing my development configuration
for a few months now, and I’m finally getting around to releasing two
of the roles I’ve created in a form (possibly) useful to other folks.

Why Ansible?

Ansible is an tool for automating system tasks like deploying packages
and configuring servers. In my case, I use it to set up cloud servers
and vagrant VMs with the packages and tools I need to be productive at
work. The ansible input files I have let me use almost any version of
an Ubuntu base image on a cloud provider to launch a disposable
development environment quickly and reliably. I no longer forget to
copy my editor’s configuration files over, and I don’t have to
remember which packages contain the tools I regularly want to use. I
can also use the ansible input files as documentation when I can’t
remember how I set something up and I want to explain it to someone
else.

Installing Ansible

Ansible is usually run on a local machine to change the configuration
of a remote machine. I configure ansible in a virtualenv on my desktop
machine, and then use it to control VMs created locally with Vagrant
or in a remote OpenStack cloud.

$ virtualenv ansible
$ source ansible/bin/activate
$ pip install ansible

Configuring Ansible

Ansible needs a configuration file to tell it where to find the server
definitions. You can also use this file to control various search paths
and other behaviors. I use the default naming, so I have an
ansible.cfg in my working directory. It contains:

[defaults]
host_key_checking = False
roles_path = roles
hostfile = hosts

Ansible manages the definition of “inventory” of available servers
separately from the “roles” those servers have. This allows you to
re-purpose servers or automatically create cloud instances and then
deploy software to them without knowing their IPs in advance.

The roles_path variable in the configuration file is a search path
for roles that might be installed globally or in a shared
directory. In order to install the roles under the current directory,
I specify a relative path to a ./roles directory.

The hostfile variable in the configuration tells ansible where to
find your servers. Advanced users can group multiple servers that
should have the same configuration, use wildcards to apply changes to
servers with similar names, etc. For this example I’m just working
with a single Vagrant VM, so my hosts file contains:

[vagrant]
192.168.30.102

[vagrant:vars]
ansible_ssh_user=vagrant
ansible_ssh_private_key_file=/Users/dhellmann/.vagrant.d/insecure_private_key

The first stanza defines a group of servers called “vagrant” with a
single member, identified by its IP address. The second stanza sets
some variables to tell ansible how to ssh into the server. The user in
the image in the VM does not match my user, so I need to specify a
name, and then I tell ansible to use vagrant’s ssh key file when
connecting as that user.

Installing Roles

I have two public roles I use for configuring my Python settings. The
python-dev role installs several versions of the Python
interpreter, including PyPy. The devpi role installs a devpi
server and configures pip to use it so repeated installations of the
same packages go more quickly.

You can check out the roles from github, or use ansible-galaxy to
install them:

$ mkdir roles
$ ansible-galaxy install dhellmann.python-dev
- downloading role 'python-dev', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-python-dev/archive/1.0.0.tar.gz
- extracting dhellmann.python-dev to roles/dhellmann.python-dev
- dhellmann.python-dev was installed successfully
$ ansible-galaxy install dhellmann.devpi
- downloading role 'devpi', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-devpi/archive/1.0.0.tar.gz
- extracting dhellmann.devpi to roles/dhellmann.devpi
- dhellmann.devpi was installed successfully

Applying the Roles

Ansible’s top-level input file is called a “playbook”. The playbook
is where you link the inventory to the roles, and provide any extra
configuration settings based on your customization needs.

The playbook to deploy both of the python developer roles to my
Vagrant VM is in a file called ansible.yml that contains:

---
- hosts: vagrant
  roles:
    - dhellmann.python-dev
    - dhellmann.devpi

The hosts specifier tells ansible to use the “vagrant” group of
servers and the roles list contains the names of the roles.

Running Ansible

Assuming the Vagrant VM is running, I can tell ansible to run the
playbook with:

$ ansible-playbook ./ansible.yml

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
changed: [192.168.30.102] => (item=python-software-properties,software-properties-common)

TASK: [dhellmann.python-dev | Install Python dependencies] ********************
changed: [192.168.30.102] => (item=libffi-dev,gcc)

TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Install python] *********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install python-dev] *****************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install easy_install from source] ***************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install pip from source] ************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install wheel] **********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Install PyPy] ***********************************
changed: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)

TASK: [dhellmann.python-dev | Install Python tools] ***************************
changed: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
changed: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Install packages] ************************************
changed: [192.168.30.102] => (item=devpi)
changed: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Install monit] ***************************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
changed: [192.168.30.102]

NOTIFIED: [dhellmann.devpi | restart monit] ***********************************
changed: [192.168.30.102]

PLAY RECAP ********************************************************************
192.168.30.102             : ok=23   changed=20   unreachable=0    failed=0

Then I can login to the VM and build some wheels:

$ vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sat Mar  7 16:46:39 EST 2015

  System load:  0.56              Users logged in:     0
  Usage of /:   25.6% of 6.98GB   IP address for eth0: 10.0.2.15
  Memory usage: 36%               IP address for eth1: 192.168.27.102
  Swap usage:   0%                IP address for eth2: 192.168.30.102
  Processes:    71

  Graph this data and manage this system at https://landscape.canonical.com/

Last login: Sat Mar  7 16:43:39 2015 from 192.168.30.1
vagrant@hubert-linuxvm:~$ pip wheel ansible
Collecting ansible
  Downloading http://localhost:3141/root/pypi/+f/e0b/bf484e9fa49cd/ansible-1.8.4.tar.gz (758kB)
    100% |################################| 761kB 13.8MB/s
Collecting paramiko (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/ea8/ad8f18a432797/paramiko-1.15.2-py2.py3-none-any.whl (165kB)
    100% |################################| 167kB 11.2MB/s
  Saved ./.pip/wheelhouse/paramiko-1.15.2-py2.py3-none-any.whl
Collecting jinja2 (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/b9d/ffd2f3b43d673/Jinja2-2.7.3.tar.gz (378kB)
    100% |################################| 380kB 17.0MB/s
Collecting PyYAML (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB)
    100% |################################| 372kB 16.3MB/s
Collecting setuptools (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/caa/7e1c8b83f4134/setuptools-14.0-py2.py3-none-any.whl (501kB)
    100% |################################| 503kB 11.9MB/s
  Saved ./.pip/wheelhouse/setuptools-14.0-py2.py3-none-any.whl
Collecting pycrypto>=2.6 (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/55a/61a054aa66812/pycrypto-2.6.1.tar.gz (446kB)
    100% |################################| 446kB 10.9MB/s
Collecting ecdsa>=0.11 (from paramiko->ansible)
  Downloading http://localhost:3141/root/pypi/+f/a32/5d50195d04959/ecdsa-0.13-py2.py3-none-any.whl (86kB)
    100% |################################| 90kB 11.1MB/s
  Saved ./.pip/wheelhouse/ecdsa-0.13-py2.py3-none-any.whl
Collecting markupsafe (from jinja2->ansible)
  Downloading http://localhost:3141/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz
Skipping paramiko, due to already being wheel.
Skipping setuptools, due to already being wheel.
Skipping ecdsa, due to already being wheel.
Building wheels for collected packages: ansible, jinja2, PyYAML, pycrypto, markupsafe
  Running setup.py bdist_wheel for ansible
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for jinja2
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for PyYAML
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for pycrypto
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for markupsafe
  Destination directory: /home/vagrant/.pip/wheelhouse
Successfully built ansible jinja2 PyYAML pycrypto markupsafe

And now the wheels are available in the local wheelhouse created by
the devpi role:

vagrant@hubert-linuxvm:~$ source /usr/local/bin/virtualenvwrapper.sh
vagrant@hubert-linuxvm:~$ mkvirtualenv ansible
New python executable in ansible/bin/python2.7
Also creating executable in ansible/bin/python
Installing setuptools, pip...done.
(ansible)vagrant@hubert-linuxvm:~$ pip install ansible
Collecting ansible
Collecting paramiko (from ansible)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.virtualenvs/ansible/lib/python2.7/site-packages (from ansible)
Collecting PyYAML (from ansible)
Collecting pycrypto>=2.6 (from ansible)
Collecting jinja2 (from ansible)
Collecting ecdsa>=0.11 (from paramiko->ansible)
Collecting markupsafe (from jinja2->ansible)
Installing collected packages: markupsafe, ecdsa, jinja2, pycrypto, PyYAML, paramiko, ansible
Successfully installed PyYAML-3.11 ansible-1.8.4 ecdsa-0.13 jinja2-2.7.3 markupsafe-0.23 paramiko-1.15.2 pycrypto-2.6.1

Passing Variables to Roles

The python-dev role supports a variable to specify which Python
versions you want installed. By default it includes 2.6, 2.7, 3.3, and
3.4 because those are the versions where I commonly want to run tests
for various packages I maintain. If you wanted only 2.7 and 3.4, you
could change the playbook to override that list of versions by adding
the python_dev_versions variable to the role specification:

---
- hosts: vagrant
  roles:
    - role: dhellmann.python-dev
      python_dev_versions:
        - version: "3.4"
        - version: "2.7"
    - dhellmann.devpi

Running ansible again will show no updates needed, but only the two
versions listed in the playbook are mentioned.

$ ansible-playbook ./ansible.yml

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
ok: [192.168.30.102] => (item=python-software-properties,software-properties-common)

TASK: [dhellmann.python-dev | Install Python dependencies] ********************
ok: [192.168.30.102] => (item=libffi-dev,gcc)

TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install python] *********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install python-dev] *****************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install easy_install from source] ***************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install pip from source] ************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install wheel] **********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install PyPy] ***********************************
ok: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)

TASK: [dhellmann.python-dev | Install Python tools] ***************************
ok: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
ok: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Install packages] ************************************
ok: [192.168.30.102] => (item=devpi)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Install monit] ***************************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
ok: [192.168.30.102]

PLAY RECAP ********************************************************************
192.168.30.102             : ok=22   changed=0    unreachable=0    failed=0

Future Work

This version of the python-dev role only works for Ubuntu, and I
would like to make it work on Fedora as well (patches welcome). I may
also add some of the system packages needed to pip install Python
libraries like lxml, since those are frequently not installed by
default.

The devpi role uses the meta-package devpi, which was
apparently deprecated as part of the most recent release. I need to
update that to install the server packages separately.