Ansible Roles for Python Developers
I have been using ansible for managing my development configuration for a few months now, and I’m finally getting around to releasing two of the roles I’ve created in a form (possibly) useful to other folks.
Why Ansible?
Ansible is an tool for automating system tasks like deploying packages and configuring servers. In my case, I use it to set up cloud servers and vagrant VMs with the packages and tools I need to be productive at work. The ansible input files I have let me use almost any version of an Ubuntu base image on a cloud provider to launch a disposable development environment quickly and reliably. I no longer forget to copy my editor’s configuration files over, and I don’t have to remember which packages contain the tools I regularly want to use. I can also use the ansible input files as documentation when I can’t remember how I set something up and I want to explain it to someone else.
Installing Ansible
Ansible is usually run on a local machine to change the configuration of a remote machine. I configure ansible in a virtualenv on my desktop machine, and then use it to control VMs created locally with Vagrant or in a remote OpenStack cloud.
$ virtualenv ansible $ source ansible/bin/activate $ pip install ansible
Configuring Ansible
Ansible needs a configuration file to tell it where to find the server
definitions. You can also use this file to control various search
paths and other behaviors. I use the default naming, so I have an
ansible.cfg in my working directory. It contains:
[defaults] host_key_checking = False roles_path = roles hostfile = hosts
Ansible manages the definition of “inventory” of available servers separately from the “roles” those servers have. This allows you to re-purpose servers or automatically create cloud instances and then deploy software to them without knowing their IPs in advance.
The roles_path variable in the configuration file is a search path
for roles that might be installed globally or in a shared
directory. In order to install the roles under the current directory,
I specify a relative path to a ./roles directory.
The hostfile variable in the configuration tells ansible where to
find your servers. Advanced users can group multiple servers that
should have the same configuration, use wildcards to apply changes to
servers with similar names, etc. For this example I’m just working
with a single Vagrant VM, so my hosts file contains:
[vagrant] 192.168.30.102 [vagrant:vars] ansible_ssh_user=vagrant ansible_ssh_private_key_file=/Users/dhellmann/.vagrant.d/insecure_private_key
The first stanza defines a group of servers called “vagrant” with a single member, identified by its IP address. The second stanza sets some variables to tell ansible how to ssh into the server. The user in the image in the VM does not match my user, so I need to specify a name, and then I tell ansible to use vagrant’s ssh key file when connecting as that user.
Installing Roles
I have two public roles I use for configuring my Python settings. The python-dev role installs several versions of the Python interpreter, including PyPy. The devpi role installs a devpi server and configures pip to use it so repeated installations of the same packages go more quickly.
You can check out the roles from github, or use ansible-galaxy to install them:
$ mkdir roles $ ansible-galaxy install dhellmann.python-dev - downloading role 'python-dev', owned by dhellmann - downloading role from https://github.com/dhellmann/ansible-python-dev/archive/1.0.0.tar.gz - extracting dhellmann.python-dev to roles/dhellmann.python-dev - dhellmann.python-dev was installed successfully $ ansible-galaxy install dhellmann.devpi - downloading role 'devpi', owned by dhellmann - downloading role from https://github.com/dhellmann/ansible-devpi/archive/1.0.0.tar.gz - extracting dhellmann.devpi to roles/dhellmann.devpi - dhellmann.devpi was installed successfully
Applying the Roles
Ansible’s top-level input file is called a “playbook”. The playbook is where you link the inventory to the roles, and provide any extra configuration settings based on your customization needs.
The playbook to deploy both of the python developer roles to my
Vagrant VM is in a file called ansible.yml that contains:
---
- hosts: vagrant
roles:
- dhellmann.python-dev
- dhellmann.devpi
The hosts specifier tells ansible to use the “vagrant” group of
servers and the roles list contains the names of the roles.
Running Ansible
Assuming the Vagrant VM is running, I can tell ansible to run the playbook with:
$ ansible-playbook ./ansible.yml
PLAY [vagrant] ****************************************************************
GATHERING FACTS ***************************************************************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
changed: [192.168.30.102] => (item=python-software-properties,software-properties-common)
TASK: [dhellmann.python-dev | Install Python dependencies] ********************
changed: [192.168.30.102] => (item=libffi-dev,gcc)
TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
changed: [192.168.30.102]
TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
changed: [192.168.30.102]
TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
changed: [192.168.30.102]
TASK: [dhellmann.python-dev | Install python] *********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install python-dev] *****************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install easy_install from source] ***************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install pip from source] ************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install wheel] **********************************
changed: [192.168.30.102] => (item={'version': '2.6'})
changed: [192.168.30.102] => (item={'version': '3.3'})
changed: [192.168.30.102] => (item={'version': '3.4'})
changed: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
changed: [192.168.30.102]
TASK: [dhellmann.python-dev | Install PyPy] ***********************************
changed: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)
TASK: [dhellmann.python-dev | Install Python tools] ***************************
changed: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
changed: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)
TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
changed: [192.168.30.102]
TASK: [dhellmann.devpi | Install packages] ************************************
changed: [192.168.30.102] => (item=devpi)
changed: [192.168.30.102] => (item=wheel)
TASK: [dhellmann.devpi | Install monit] ***************************************
changed: [192.168.30.102]
TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
changed: [192.168.30.102]
TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
changed: [192.168.30.102]
TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
changed: [192.168.30.102]
NOTIFIED: [dhellmann.devpi | restart monit] ***********************************
changed: [192.168.30.102]
PLAY RECAP ********************************************************************
192.168.30.102 : ok=23 changed=20 unreachable=0 failed=0
Then I can login to the VM and build some wheels:
$ vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Sat Mar 7 16:46:39 EST 2015
System load: 0.56 Users logged in: 0
Usage of /: 25.6% of 6.98GB IP address for eth0: 10.0.2.15
Memory usage: 36% IP address for eth1: 192.168.27.102
Swap usage: 0% IP address for eth2: 192.168.30.102
Processes: 71
Graph this data and manage this system at https://landscape.canonical.com/
Last login: Sat Mar 7 16:43:39 2015 from 192.168.30.1
vagrant@hubert-linuxvm:~$ pip wheel ansible
Collecting ansible
Downloading http://localhost:3141/root/pypi/+f/e0b/bf484e9fa49cd/ansible-1.8.4.tar.gz (758kB)
100% |################################| 761kB 13.8MB/s
Collecting paramiko (from ansible)
Downloading http://localhost:3141/root/pypi/+f/ea8/ad8f18a432797/paramiko-1.15.2-py2.py3-none-any.whl (165kB)
100% |################################| 167kB 11.2MB/s
Saved ./.pip/wheelhouse/paramiko-1.15.2-py2.py3-none-any.whl
Collecting jinja2 (from ansible)
Downloading http://localhost:3141/root/pypi/+f/b9d/ffd2f3b43d673/Jinja2-2.7.3.tar.gz (378kB)
100% |################################| 380kB 17.0MB/s
Collecting PyYAML (from ansible)
Downloading http://localhost:3141/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB)
100% |################################| 372kB 16.3MB/s
Collecting setuptools (from ansible)
Downloading http://localhost:3141/root/pypi/+f/caa/7e1c8b83f4134/setuptools-14.0-py2.py3-none-any.whl (501kB)
100% |################################| 503kB 11.9MB/s
Saved ./.pip/wheelhouse/setuptools-14.0-py2.py3-none-any.whl
Collecting pycrypto>=2.6 (from ansible)
Downloading http://localhost:3141/root/pypi/+f/55a/61a054aa66812/pycrypto-2.6.1.tar.gz (446kB)
100% |################################| 446kB 10.9MB/s
Collecting ecdsa>=0.11 (from paramiko->ansible)
Downloading http://localhost:3141/root/pypi/+f/a32/5d50195d04959/ecdsa-0.13-py2.py3-none-any.whl (86kB)
100% |################################| 90kB 11.1MB/s
Saved ./.pip/wheelhouse/ecdsa-0.13-py2.py3-none-any.whl
Collecting markupsafe (from jinja2->ansible)
Downloading http://localhost:3141/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz
Skipping paramiko, due to already being wheel.
Skipping setuptools, due to already being wheel.
Skipping ecdsa, due to already being wheel.
Building wheels for collected packages: ansible, jinja2, PyYAML, pycrypto, markupsafe
Running setup.py bdist_wheel for ansible
Destination directory: /home/vagrant/.pip/wheelhouse
Running setup.py bdist_wheel for jinja2
Destination directory: /home/vagrant/.pip/wheelhouse
Running setup.py bdist_wheel for PyYAML
Destination directory: /home/vagrant/.pip/wheelhouse
Running setup.py bdist_wheel for pycrypto
Destination directory: /home/vagrant/.pip/wheelhouse
Running setup.py bdist_wheel for markupsafe
Destination directory: /home/vagrant/.pip/wheelhouse
Successfully built ansible jinja2 PyYAML pycrypto markupsafe
And now the wheels are available in the local wheelhouse created by the devpi role:
vagrant@hubert-linuxvm:~$ source /usr/local/bin/virtualenvwrapper.sh vagrant@hubert-linuxvm:~$ mkvirtualenv ansible New python executable in ansible/bin/python2.7 Also creating executable in ansible/bin/python Installing setuptools, pip...done. (ansible)vagrant@hubert-linuxvm:~$ pip install ansible Collecting ansible Collecting paramiko (from ansible) Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.virtualenvs/ansible/lib/python2.7/site-packages (from ansible) Collecting PyYAML (from ansible) Collecting pycrypto>=2.6 (from ansible) Collecting jinja2 (from ansible) Collecting ecdsa>=0.11 (from paramiko->ansible) Collecting markupsafe (from jinja2->ansible) Installing collected packages: markupsafe, ecdsa, jinja2, pycrypto, PyYAML, paramiko, ansible Successfully installed PyYAML-3.11 ansible-1.8.4 ecdsa-0.13 jinja2-2.7.3 markupsafe-0.23 paramiko-1.15.2 pycrypto-2.6.1
Passing Variables to Roles
The python-dev role supports a variable to specify which Python
versions you want installed. By default it includes 2.6, 2.7, 3.3, and
3.4 because those are the versions where I commonly want to run tests
for various packages I maintain. If you wanted only 2.7 and 3.4, you
could change the playbook to override that list of versions by adding
the python_dev_versions variable to the role specification:
---
- hosts: vagrant
roles:
- role: dhellmann.python-dev
python_dev_versions:
- version: "3.4"
- version: "2.7"
- dhellmann.devpi
Running ansible again will show no updates needed, but only the two versions listed in the playbook are mentioned.
$ ansible-playbook ./ansible.yml
PLAY [vagrant] ****************************************************************
GATHERING FACTS ***************************************************************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Install tools for adding PPA repositories] ******
ok: [192.168.30.102] => (item=python-software-properties,software-properties-common)
TASK: [dhellmann.python-dev | Install Python dependencies] ********************
ok: [192.168.30.102] => (item=libffi-dev,gcc)
TASK: [dhellmann.python-dev | Add Python 2.6 repository] **********************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Remove system package for pip] ******************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Download ez_setup.py] ***************************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Download get-pip.py] ****************************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Install python] *********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install python-dev] *****************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install easy_install from source] ***************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install pip from source] ************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Install wheel] **********************************
ok: [192.168.30.102] => (item={'version': '3.4'})
ok: [192.168.30.102] => (item={'version': '2.7'})
TASK: [dhellmann.python-dev | Add PyPy PPA repository] ************************
ok: [192.168.30.102]
TASK: [dhellmann.python-dev | Install PyPy] ***********************************
ok: [192.168.30.102] => (item=pypy,pypy-dev,python-cffi)
TASK: [dhellmann.python-dev | Install Python tools] ***************************
ok: [192.168.30.102] => (item=tox)
ok: [192.168.30.102] => (item=virtualenv)
ok: [192.168.30.102] => (item=virtualenvwrapper)
ok: [192.168.30.102] => (item=wheel)
TASK: [dhellmann.devpi | Create devpi virtualenv] *****************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | Install packages] ************************************
ok: [192.168.30.102] => (item=devpi)
ok: [192.168.30.102] => (item=wheel)
TASK: [dhellmann.devpi | Install monit] ***************************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | Make sure monit is running] **************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | Configure monit to run devpi] ************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | Create ~/.pip/wheelhouse] ****************************
ok: [192.168.30.102]
TASK: [dhellmann.devpi | ~/.pip.conf] *****************************************
ok: [192.168.30.102]
PLAY RECAP ********************************************************************
192.168.30.102 : ok=22 changed=0 unreachable=0 failed=0
Future Work
This version of the python-dev role only works for Ubuntu, and I
would like to make it work on Fedora as well (patches welcome). I may
also add some of the system packages needed to pip install Python
libraries like lxml, since those are frequently not installed by
default.
The devpi role uses the meta-package devpi, which was apparently
deprecated as part of the most recent release. I need to update that
to install the server packages separately.