Ansible Roles for Python Developers

I have been using ansible for managing my development configuration
for a few months now, and I’m finally getting around to releasing two
of the roles I’ve created in a form (possibly) useful to other folks.

Why Ansible?

Ansible is an tool for automating system tasks like deploying packages
and configuring servers. In my case, I use it to set up cloud servers
and vagrant VMs with the packages and tools I need to be productive at
work. The ansible input files I have let me use almost any version of
an Ubuntu base image on a cloud provider to launch a disposable
development environment quickly and reliably. I no longer forget to
copy my editor’s configuration files over, and I don’t have to
remember which packages contain the tools I regularly want to use. I
can also use the ansible input files as documentation when I can’t
remember how I set something up and I want to explain it to someone
else.

Installing Ansible

Ansible is usually run on a local machine to change the configuration
of a remote machine. I configure ansible in a virtualenv on my desktop
machine, and then use it to control VMs created locally with Vagrant
or in a remote OpenStack cloud.

$ virtualenv ansible
$ source ansible/bin/activate
$ pip install ansible

Configuring Ansible

Ansible needs a configuration file to tell it where to find the server
definitions. You can also use this file to control various search paths
and other behaviors. I use the default naming, so I have an
ansible.cfg in my working directory. It contains:

[defaults]
host_key_checking = False
roles_path = roles
hostfile = hosts

Ansible manages the definition of “inventory” of available servers
separately from the “roles” those servers have. This allows you to
re-purpose servers or automatically create cloud instances and then
deploy software to them without knowing their IPs in advance.

The roles_path variable in the configuration file is a search path
for roles that might be installed globally or in a shared
directory. In order to install the roles under the current directory,
I specify a relative path to a ./roles directory.

The hostfile variable in the configuration tells ansible where to
find your servers. Advanced users can group multiple servers that
should have the same configuration, use wildcards to apply changes to
servers with similar names, etc. For this example I’m just working
with a single Vagrant VM, so my hosts file contains:

[vagrant]
192.168.30.102

[vagrant:vars]
ansible_ssh_user=vagrant
ansible_ssh_private_key_file=/Users/dhellmann/.vagrant.d/insecure_private_key

The first stanza defines a group of servers called “vagrant” with a
single member, identified by its IP address. The second stanza sets
some variables to tell ansible how to ssh into the server. The user in
the image in the VM does not match my user, so I need to specify a
name, and then I tell ansible to use vagrant’s ssh key file when
connecting as that user.

Installing Roles

I have two public roles I use for configuring my Python settings. The
python-dev role installs several versions of the Python
interpreter, including PyPy. The devpi role installs a devpi
server and configures pip to use it so repeated installations of the
same packages go more quickly.

You can check out the roles from github, or use ansible-galaxy to
install them:

$ mkdir roles
$ ansible-galaxy install dhellmann.python-dev
- downloading role 'python-dev', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-python-dev/archive/1.0.0.tar.gz
- extracting dhellmann.python-dev to roles/dhellmann.python-dev
- dhellmann.python-dev was installed successfully
$ ansible-galaxy install dhellmann.devpi
- downloading role 'devpi', owned by dhellmann
- downloading role from https://github.com/dhellmann/ansible-devpi/archive/1.0.0.tar.gz
- extracting dhellmann.devpi to roles/dhellmann.devpi
- dhellmann.devpi was installed successfully

Applying the Roles

Ansible’s top-level input file is called a “playbook”. The playbook
is where you link the inventory to the roles, and provide any extra
configuration settings based on your customization needs.

The playbook to deploy both of the python developer roles to my
Vagrant VM is in a file called ansible.yml that contains:

---
- hosts: vagrant
  roles:
    - dhellmann.python-dev
    - dhellmann.devpi

The hosts specifier tells ansible to use the “vagrant” group of
servers and the roles list contains the names of the roles.

Running Ansible

Assuming the Vagrant VM is running, I can tell ansible to run the
playbook with:

$ ansible-playbook ./ansible.yml

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
changed: [192.168.30.102] => (item=python-software-properties,software-properties-common)

TASK: [dhellmann.python-dev | Install Python dependencies] ********************
changed: [192.168.30.102] => (item=libffi-dev,gcc)

TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Install python] *********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install python-dev] *****************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install easy_install from source] ***************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install pip from source] ************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install wheel] **********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
changed: [192.168.30.102]

TASK: [dhellmann.python-dev | Install PyPy] ***********************************
changed: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)

TASK: [dhellmann.python-dev | Install Python tools] ***************************
changed: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
changed: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Install packages] ************************************
changed: [192.168.30.102] => (item=devpi)
changed: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Install monit] ***************************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
changed: [192.168.30.102]

TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
changed: [192.168.30.102]

NOTIFIED: [dhellmann.devpi | restart monit] ***********************************
changed: [192.168.30.102]

PLAY RECAP ********************************************************************
192.168.30.102             : ok=23   changed=20   unreachable=0    failed=0

Then I can login to the VM and build some wheels:

$ vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sat Mar  7 16:46:39 EST 2015

  System load:  0.56              Users logged in:     0
  Usage of /:   25.6% of 6.98GB   IP address for eth0: 10.0.2.15
  Memory usage: 36%               IP address for eth1: 192.168.27.102
  Swap usage:   0%                IP address for eth2: 192.168.30.102
  Processes:    71

  Graph this data and manage this system at https://landscape.canonical.com/

Last login: Sat Mar  7 16:43:39 2015 from 192.168.30.1
vagrant@hubert-linuxvm:~$ pip wheel ansible
Collecting ansible
  Downloading http://localhost:3141/root/pypi/+f/e0b/bf484e9fa49cd/ansible-1.8.4.tar.gz (758kB)
    100% |################################| 761kB 13.8MB/s
Collecting paramiko (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/ea8/ad8f18a432797/paramiko-1.15.2-py2.py3-none-any.whl (165kB)
    100% |################################| 167kB 11.2MB/s
  Saved ./.pip/wheelhouse/paramiko-1.15.2-py2.py3-none-any.whl
Collecting jinja2 (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/b9d/ffd2f3b43d673/Jinja2-2.7.3.tar.gz (378kB)
    100% |################################| 380kB 17.0MB/s
Collecting PyYAML (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB)
    100% |################################| 372kB 16.3MB/s
Collecting setuptools (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/caa/7e1c8b83f4134/setuptools-14.0-py2.py3-none-any.whl (501kB)
    100% |################################| 503kB 11.9MB/s
  Saved ./.pip/wheelhouse/setuptools-14.0-py2.py3-none-any.whl
Collecting pycrypto>=2.6 (from ansible)
  Downloading http://localhost:3141/root/pypi/+f/55a/61a054aa66812/pycrypto-2.6.1.tar.gz (446kB)
    100% |################################| 446kB 10.9MB/s
Collecting ecdsa>=0.11 (from paramiko->ansible)
  Downloading http://localhost:3141/root/pypi/+f/a32/5d50195d04959/ecdsa-0.13-py2.py3-none-any.whl (86kB)
    100% |################################| 90kB 11.1MB/s
  Saved ./.pip/wheelhouse/ecdsa-0.13-py2.py3-none-any.whl
Collecting markupsafe (from jinja2->ansible)
  Downloading http://localhost:3141/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz
Skipping paramiko, due to already being wheel.
Skipping setuptools, due to already being wheel.
Skipping ecdsa, due to already being wheel.
Building wheels for collected packages: ansible, jinja2, PyYAML, pycrypto, markupsafe
  Running setup.py bdist_wheel for ansible
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for jinja2
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for PyYAML
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for pycrypto
  Destination directory: /home/vagrant/.pip/wheelhouse
  Running setup.py bdist_wheel for markupsafe
  Destination directory: /home/vagrant/.pip/wheelhouse
Successfully built ansible jinja2 PyYAML pycrypto markupsafe

And now the wheels are available in the local wheelhouse created by
the devpi role:

vagrant@hubert-linuxvm:~$ source /usr/local/bin/virtualenvwrapper.sh
vagrant@hubert-linuxvm:~$ mkvirtualenv ansible
New python executable in ansible/bin/python2.7
Also creating executable in ansible/bin/python
Installing setuptools, pip...done.
(ansible)vagrant@hubert-linuxvm:~$ pip install ansible
Collecting ansible
Collecting paramiko (from ansible)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.virtualenvs/ansible/lib/python2.7/site-packages (from ansible)
Collecting PyYAML (from ansible)
Collecting pycrypto>=2.6 (from ansible)
Collecting jinja2 (from ansible)
Collecting ecdsa>=0.11 (from paramiko->ansible)
Collecting markupsafe (from jinja2->ansible)
Installing collected packages: markupsafe, ecdsa, jinja2, pycrypto, PyYAML, paramiko, ansible
Successfully installed PyYAML-3.11 ansible-1.8.4 ecdsa-0.13 jinja2-2.7.3 markupsafe-0.23 paramiko-1.15.2 pycrypto-2.6.1

Passing Variables to Roles

The python-dev role supports a variable to specify which Python
versions you want installed. By default it includes 2.6, 2.7, 3.3, and
3.4 because those are the versions where I commonly want to run tests
for various packages I maintain. If you wanted only 2.7 and 3.4, you
could change the playbook to override that list of versions by adding
the python_dev_versions variable to the role specification:

---
- hosts: vagrant
  roles:
    - role: dhellmann.python-dev
      python_dev_versions:
        - version: "3.4"
        - version: "2.7"
    - dhellmann.devpi

Running ansible again will show no updates needed, but only the two
versions listed in the playbook are mentioned.

$ ansible-playbook ./ansible.yml

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
ok: [192.168.30.102] => (item=python-software-properties,software-properties-common)

TASK: [dhellmann.python-dev | Install Python dependencies] ********************
ok: [192.168.30.102] => (item=libffi-dev,gcc)

TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install python] *********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install python-dev] *****************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install easy_install from source] ***************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install pip from source] ************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Install wheel] **********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})

TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
ok: [192.168.30.102]

TASK: [dhellmann.python-dev | Install PyPy] ***********************************
ok: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)

TASK: [dhellmann.python-dev | Install Python tools] ***************************
ok: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
ok: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Install packages] ************************************
ok: [192.168.30.102] => (item=devpi)
ok: [192.168.30.102] => (item=wheel)

TASK: [dhellmann.devpi | Install monit] ***************************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
ok: [192.168.30.102]

TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
ok: [192.168.30.102]

PLAY RECAP ********************************************************************
192.168.30.102             : ok=22   changed=0    unreachable=0    failed=0

Future Work

This version of the python-dev role only works for Ubuntu, and I
would like to make it work on Fedora as well (patches welcome). I may
also add some of the system packages needed to pip install Python
libraries like lxml, since those are frequently not installed by
default.

The devpi role uses the meta-package devpi, which was
apparently deprecated as part of the most recent release. I need to
update that to install the server packages separately.