virtualenvwrapper

Not content to leave well enough alone, Doug offers up some
extensions to Ian Bicking’s virtualenv script that make it even
more useful.

Back in February, I introduced Ian Bicking’s virtualenv script for creating
isolated Python environments, complete with their own interpreter and
site-packages directory. I’ve been using virtualenv for all of my
projects since then, but keeping up with all of the environments I
have been creating has turned a little messy. I’ve solved the problem
by writing a few wrappers in the
form of bash functions. If you are a virtualenv user, you may find
them useful, too.

virtualenv

As I mentioned in the previous column, bringing you interesting
articles about new projects from the far reaches of the internet means
that I end up installing a lot of Python packages to review code
before it is published. That’s okay, but I want my system to be
usable for working on my own development projects, too. Of course,
creating a separate “sandbox” for each article or project means I can
install whatever libraries I need, and not worry about version
conflicts or deleting something accidentally. It’s almost like Ian
wrote virtualenv specifically for me.

For new subscribers who missed the February column, here’s a quick
primer in virtualenv: Every time you run virtualenv, it sets up a
clean copy of Python in a new directory by copying or linking files
from your primary Python installation to create new bin and
lib directories. To use the sandbox, you simply run source
$path/bin/activate
. Your environment is then reconfigured so that
the version of Python in the virtual environment takes precedent over
the normal version installed globally on the system. Since you
created the directory yourself, you have all the permissions you need
to install new modules or libraries into the environment. And since
the environments are light weight and easy to create, you can have as
many of them as you want.

Too Many Virtual Environments

So far, so good. An average user would probably create a new sandbox
for each ongoing project, with some temporary environments thrown in
once in a while for testing a new version of a library or playing with
something discovered through the Python Package Index feed. I, on the
other hand, found myself creating 4-6 new environments every few weeks
as I created a separate environment for each article being reviewed
for the magazine. The number of environments I had quickly grew to be
unmanageable.

I started with all of my virtual environments in the same directories
that held the other files for the article, so I could easily find the
activate script used to enable the environment. This way of
working meant I had environments scattered all through out my
Documents folder, though, and I had to cd to each directory to
make sure I was looking at the right environment before activating it.
Another complication was the temporary environments were mixed
together with “real” files that made up the articles and other parts
of the magazine. Since we use version control to manage those files,
I had to constantly tell the version control tools to ignore the
virtual environment files.

I decided the first change I needed to make was to just put all of the
environments together in one place, so I could see them all easily and
select the right one quickly. I created a directory,
~/.virtualenvs to hold all of my new virtual environments, and
moved my existing environments there. I gave each one a name based on
the project or article it was associated with to make it easy to tell
them apart.

To create a new environment, all I had to do was:

$ cd ~/.virtualenvs
$ virtualenv newname
$ cd -

which I eventually shortened to just:

$ (cd ~/.virtualenvs; virtualenv newname)

I type pretty quickly, but I’m still averse to repeating myself. The
only part of the command that changed each time I made a new
environment was the newname argument to virtualenv. It looked
like a perfect opportunity to use a wrapper to streamline the command.
I thought about using a shell alias, but my bash alias-fu isn’t all
that strong and I wanted to include some basic error checking.
Creating a shell function however, was very easy, and combined all
of the benefits of an alias with the syntax of a script.

As these “simple” projects tend to do, this one quickly grew to
include a few more features. Eventually it reached the point where my
desire to hack on it was satisfied and I could actually use it. The
results, called virtualenvwrapper, are available in Listing 1.

Managing Environments

The bash script in Listing 1 is intended to be run during your shell
login sequence via the source command. The distributed version of
the file is (unimaginatively) named virtualenvwrapper_bashrc, so
to use it you would add a line like this to the end of your
~/.bashrc file:

source $HOME/bin/virtualenvwrapper_bashrc

Of course the actual path depends on where you put the file when you
download it. There is just the one file, so I keep a copy in
$HOME/bin, but you might have other standard practices for
extension scripts like this one.

The only other setup steps you need to take before using the wrapper
is to define the shell variable WORKON_HOME to point to the
directory where your virtual environments will go. The default value
is ~/.virtualenvs, and you don’t have to change it if you want to
put them somewhere else. You can place the setting before or after
you source the script, so you would end up with something like this:

export WORKON_HOME="$HOME/Devel/Environments"
source $HOME/bin/virtualenvwrapper_bashrc

Once you have both commands in your login script, just source
~/.bashrc
and then you can start creating
environments. mkvirtualenv is a very thin wrapper around
virtualenv itself that creates an environment and then immediately
activates it.

$ mkvirtualenv newenv
New python executable in newenv/bin/python
Installing setuptools....................done.
(newenv)farnsworth:dhellmann:~:502 $

In fact, all of the arguments you give to mkvirtualenv are passed
directly to virtualenv, so you can use -h, –no-site-packages,
and all of the other normal options.

Switching Environments

One of the primary reasons I created virtualenvwrapper was to make it
easier for me to alternate between different environments from the
same shell. The workon function is the interface for switching,
named that way because I use it when I want to “work on” a different
project. To see a list of available environments, run workon with
no arguments. To switch to an environment, give the name as an
argument.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
(newenv)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
newenv
personal
pymag
pymotw

As you see here, most of the available environments are for my own
personal projects. You may recognize PyGameTutorial as being the
environment I have been using to test Terry Hancock’s code for his
tutorial series, the first installment of which appears elsewhere in
this issue. To switch to that environment, I simply run workon
PyGameTutorial
:

(newenv)$ workon PyGameTutorial
(PyGameTutorial)$

To switch to the sandbox I use for writing the “Python Module of the
Week”, I would run:

(PyGameTutorial)$ workon pymotw
(pymotw)$

Fancy Switching

Sometimes, as with the PyMOTW series, I want to change more about the
shell environment than just the virtual environment settings. For
example, I have a working directory where I create all of the code for
PyMOTW, and when I “switch” to working on it, I want my shell to go
there. To allow those sorts of actions, I added a feature to
workon to look for hook scripts inside the $VIRTUAL_ENV/bin
directory, and run them before and after activating the environment.

Before you move out of an environment, you usually want to save the
project in your editor, clean up temporary files, and otherwise
preserve its current state. The predeactivate hook offers an
opportunity to clean up the current environment, before switching to
the new one.

If the postactivate script is present, it is run after the new
environment is configured. So to do any extra customization for an
environment, you can simply create the postactivate script with
the commands and they will be run automatically. You might want to
open a new project file in your editor, set your terminal window
title, or take any number of other customization steps.

Both of the hook scripts are sourced in the current shell, rather than
being run as a new program, to give them an opportunity to change
active environment variables. You can use them to change your
PATH or re-configure your shell prompt, for example. Anything you
can do from the shell command line can be done in the pre and
post activation scripts.

In the case of PyMOTW, I have a simple call to cd in the
postactivate script. I include a similar command in the
postactivate script for each temporary environment I create for a
new article, to make it easy to move around within the magazine source
directories.

Cleaning Up

The problem that led me to create these shell functions in the first
place was that I had too many virtual environments floating around my
hard drive. At this point I have described tools I wrote to make
creating new environments even easier. I also added a simple
function, rmvirtualenv to make removing environments safe and
easy.

Removing an environment is simple with rm, but after accidentally
removing my working environment once, I thought the extra check was
worth a new command. rmvirtualenv lets you remove any environment
by name, as long as it is not the currently activated environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
(pymotw)$ rmvirtualenv pymotw
ERROR: You cannot remove the active environment.
(pymotw)$ rmvirtualenv newenv
(pymotw)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
personal
pymag
pymotw

Conclusions

Some readers may wonder why I gave the shell functions
mkvirtualenv and rmvirtualenv such long names if I was trying
to avoid doing so much typing. Using tab completion at the shell
prompt, I never have to type the whole command. mkvirt or
rmvirt followed by TAB expands to the complete command name.

Although virtualenvwrapper meets my needs today, there are a few more
features I would like to add, eventually. No software is ever
actually done, right?

rmvirtualenv really needs another safety switch to make sure the
user means it. It could, for example, list all of the modules or eggs
installed in the private site-packages directory and then prompt
the user to make sure they really want to remove the environment.

workon should offer a “none” option, to clear the virtual
environment settings from the current shell without switching to
another one. For now I work around that by having an empty “2.5”
environment that mirrors my installed site-packages.

As I have said before, virtualenv has become an integral part of my
toolbox. Whether for my own projects or editorial reviews, the
ability to create a clean scratch area where I can install software
with impunity gives me a sense of freedom that working directly out of
/usr/local just can’t match. I hope you give virtualenv a try,
and if it suits you, maybe virtualenvwrapper will be a useful addition
as well.

Next Month

Next month this series will continue with coverage of more tools to
enhance your programming productivity. If you have a tip to share,
feedback on something I’ve written, or if there is a topic you would
like for me to cover in this column, send a note with the details to
doug dot hellmann at pythonmagazine dot com and let me know, or add
the link to your del.icio.us account with the tag pymagdifferent.

Originally published in Python Magazine Volume 2 Issue 5 , May, 2008