virtualenvwrapper
Not content to leave well enough alone, Doug offers up some extensions to Ian Bicking’s virtualenv script that make it even more useful.
Back in February, I introduced Ian Bicking’s virtualenv
script for creating isolated Python environments, complete with their
own interpreter and site-packages directory. I’ve been using
virtualenv for all of my projects since then, but keeping up with all
of the environments I have been creating has turned a little
messy. I’ve solved the problem by writing a few
wrappers in the
form of bash functions. If you are a virtualenv user, you may find
them useful, too.
virtualenv
As I mentioned in the previous column, bringing you interesting articles about new projects from the far reaches of the internet means that I end up installing a lot of Python packages to review code before it is published. That’s okay, but I want my system to be usable for working on my own development projects, too. Of course, creating a separate “sandbox” for each article or project means I can install whatever libraries I need, and not worry about version conflicts or deleting something accidentally. It’s almost like Ian wrote virtualenv specifically for me.
For new subscribers who missed the February column, here’s a quick primer in virtualenv: Every time you run virtualenv, it sets up a clean copy of Python in a new directory by copying or linking files from your primary Python installation to create new bin and lib directories. To use the sandbox, you simply run source $path/bin/activate. Your environment is then reconfigured so that the version of Python in the virtual environment takes precedent over the normal version installed globally on the system. Since you created the directory yourself, you have all the permissions you need to install new modules or libraries into the environment. And since the environments are light weight and easy to create, you can have as many of them as you want.
Too Many Virtual Environments
So far, so good. An average user would probably create a new sandbox for each ongoing project, with some temporary environments thrown in once in a while for testing a new version of a library or playing with something discovered through the Python Package Index feed. I, on the other hand, found myself creating 4-6 new environments every few weeks as I created a separate environment for each article being reviewed for the magazine. The number of environments I had quickly grew to be unmanageable.
I started with all of my virtual environments in the same directories that held the other files for the article, so I could easily find the activate script used to enable the environment. This way of working meant I had environments scattered all through out my Documents folder, though, and I had to cd to each directory to make sure I was looking at the right environment before activating it. Another complication was the temporary environments were mixed together with “real” files that made up the articles and other parts of the magazine. Since we use version control to manage those files, I had to constantly tell the version control tools to ignore the virtual environment files.
I decided the first change I needed to make was to just put all of the environments together in one place, so I could see them all easily and select the right one quickly. I created a directory, ~/.virtualenvs to hold all of my new virtual environments, and moved my existing environments there. I gave each one a name based on the project or article it was associated with to make it easy to tell them apart.
To create a new environment, all I had to do was:
$ cd ~/.virtualenvs
$ virtualenv newname
$ cd -
which I eventually shortened to just:
$ (cd ~/.virtualenvs; virtualenv newname)
I type pretty quickly, but I’m still averse to repeating myself. The only part of the command that changed each time I made a new environment was the newname argument to virtualenv. It looked like a perfect opportunity to use a wrapper to streamline the command. I thought about using a shell alias, but my bash alias-fu isn’t all that strong and I wanted to include some basic error checking. Creating a shell function however, was very easy, and combined all of the benefits of an alias with the syntax of a script.
As these “simple” projects tend to do, this one quickly grew to include a few more features. Eventually it reached the point where my desire to hack on it was satisfied and I could actually use it. The results, called virtualenvwrapper, are available in Listing 1.
Managing Environments
The bash script in Listing 1 is intended to be run during your shell login sequence via the source command. The distributed version of the file is (unimaginatively) named virtualenvwrapper_bashrc, so to use it you would add a line like this to the end of your ~/.bashrc file:
source $HOME/bin/virtualenvwrapper_bashrc
Of course the actual path depends on where you put the file when you download it. There is just the one file, so I keep a copy in $HOME/bin, but you might have other standard practices for extension scripts like this one.
The only other setup steps you need to take before using the wrapper is to define the shell variable WORKON_HOME to point to the directory where your virtual environments will go. The default value is ~/.virtualenvs, and you don’t have to change it if you want to put them somewhere else. You can place the setting before or after you source the script, so you would end up with something like this:
export WORKON_HOME="$HOME/Devel/Environments"
source $HOME/bin/virtualenvwrapper_bashrc
Once you have both commands in your login script, just source ~/.bashrc and then you can start creating environments. mkvirtualenv is a very thin wrapper around virtualenv itself that creates an environment and then immediately activates it.
$ mkvirtualenv newenv
New python executable in newenv/bin/python
Installing setuptools....................done.
(newenv)farnsworth:dhellmann:~:502 $
In fact, all of the arguments you give to mkvirtualenv are passed directly to virtualenv, so you can use -h, --no-site-packages, and all of the other normal options.
Switching Environments
One of the primary reasons I created virtualenvwrapper was to make it easier for me to alternate between different environments from the same shell. The workon function is the interface for switching, named that way because I use it when I want to “work on” a different project. To see a list of available environments, run workon with no arguments. To switch to an environment, give the name as an argument.
(newenv)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
newenv
personal
pymag
pymotw
As you see here, most of the available environments are for my own personal projects. You may recognize PyGameTutorial as being the environment I have been using to test Terry Hancock’s code for his tutorial series, the first installment of which appears elsewhere in this issue. To switch to that environment, I simply run workon PyGameTutorial:
(newenv)$ workon PyGameTutorial
(PyGameTutorial)$
To switch to the sandbox I use for writing the “Python Module of the Week”, I would run:
(PyGameTutorial)$ workon pymotw
(pymotw)$
Fancy Switching
Sometimes, as with the PyMOTW series, I want to change more about the shell environment than just the virtual environment settings. For example, I have a working directory where I create all of the code for PyMOTW, and when I “switch” to working on it, I want my shell to go there. To allow those sorts of actions, I added a feature to workon to look for hook scripts inside the $VIRTUAL_ENV/bin directory, and run them before and after activating the environment.
Before you move out of an environment, you usually want to save the project in your editor, clean up temporary files, and otherwise preserve its current state. The predeactivate hook offers an opportunity to clean up the current environment, before switching to the new one.
If the postactivate script is present, it is run after the new environment is configured. So to do any extra customization for an environment, you can simply create the postactivate script with the commands and they will be run automatically. You might want to open a new project file in your editor, set your terminal window title, or take any number of other customization steps.
Both of the hook scripts are sourced in the current shell, rather than being run as a new program, to give them an opportunity to change active environment variables. You can use them to change your PATH or re-configure your shell prompt, for example. Anything you can do from the shell command line can be done in the pre and post activation scripts.
In the case of PyMOTW, I have a simple call to cd in the postactivate script. I include a similar command in the postactivate script for each temporary environment I create for a new article, to make it easy to move around within the magazine source directories.
Cleaning Up
The problem that led me to create these shell functions in the first place was that I had too many virtual environments floating around my hard drive. At this point I have described tools I wrote to make creating new environments even easier. I also added a simple function, rmvirtualenv to make removing environments safe and easy.
Removing an environment is simple with rm, but after accidentally removing my working environment once, I thought the extra check was worth a new command. rmvirtualenv lets you remove any environment by name, as long as it is not the currently activated environment.
(pymotw)$ rmvirtualenv pymotw
ERROR: You cannot remove the active environment.
(pymotw)$ rmvirtualenv newenv
(pymotw)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
personal
pymag
pymotw
Conclusions
Some readers may wonder why I gave the shell functions mkvirtualenv and rmvirtualenv such long names if I was trying to avoid doing so much typing. Using tab completion at the shell prompt, I never have to type the whole command. mkvirt or rmvirt followed by TAB expands to the complete command name.
Although virtualenvwrapper meets my needs today, there are a few more features I would like to add, eventually. No software is ever actually done, right?
rmvirtualenv really needs another safety switch to make sure the user means it. It could, for example, list all of the modules or eggs installed in the private site-packages directory and then prompt the user to make sure they really want to remove the environment.
workon should offer a “none” option, to clear the virtual environment settings from the current shell without switching to another one. For now I work around that by having an empty “2.5” environment that mirrors my installed site-packages.
As I have said before, virtualenv has become an integral part of my toolbox. Whether for my own projects or editorial reviews, the ability to create a clean scratch area where I can install software with impunity gives me a sense of freedom that working directly out of /usr/local just can’t match. I hope you give virtualenv a try, and if it suits you, maybe virtualenvwrapper will be a useful addition as well.
Next Month
Next month this series will continue with coverage of more tools to enhance your programming productivity. If you have a tip to share, feedback on something I’ve written, or if there is a topic you would like for me to cover in this column, send a note with the details to doug dot hellmann at pythonmagazine dot com and let me know, or add the link to your del.icio.us account with the tag pymagdifferent.