holiday coding and virtualenvwrapper updates

In between other holiday-related activities I have had some time to
work on some personal projects a bit over the past week. The result
includes new releases of virtualenvwrapper, virtualenvwrapper.project,
and a new product called sphinxcontrib-spelling.

virtualenvwrapper updates

Version 2.6.1 of virtualenvwrapper includes a few fixes to make it
work better under Python 2.4 and correct some problems with Cygwin. It
also includes a major overhaul of the testing infrastructure to use
tox instead of the home-grown scripts I was using for testing with
multiple versions of Python. The tests now work reliably under Python
2.4 – 2.7 with bash, zsh, and ksh. And, more importantly, they run on
systems other than my laptop, so that other developers can run the
tests (and add new ones) before contributing patches.

The latest version of virtualenvwrapper.project (the extension that
adds project management to virtualenvwrapper) has had similar testing
updates made. I also added a cdproject command to return your shell to
the project home directory easily.

Spelling Checker for Sphinx

Finally, I started a new project to integrate PyEnchant with
Sphinx to make it easy to check the spelling in large
reStructuredText documents that span many files. The result,
sphinxcontrib-spelling, is in beta right now. I have some additional
features that I want to add, but it is usable in this form and ready for
feedback from users. I am working on a more extensive write-up of
creating a Sphinx extension like this, to be posted soon.

more virtualenvwrapper enhancements

Yesterday I released version 2.1 of virtualenvwrapper. The primary
purpose of this release is a set of enhancements to support
virtualenvwrapper.project, a new extension to manage project
work directories with templates.

The base project plugin gives you automatic creation of work
directories to hold sources and other artifacts for a project. Each time
you run mkproject you’ll get a virtualenv and a matching directory in
$PROJECT_HOME. Continuing the theme of extensibility, there is also a
new API for creating project “templates”. A template has complete access
to the virtualenv and project directories, and can install software,
configure it, set up source repositories, etc. Anything you need to do
on a regular basis when starting work on a new project.

To illustrate the utility of project templates, I also released
virtualenvwrapper.bitbucket, a template to automatically clone a
Mercurial repository from BitBucket each time a new project is created.

Here’s an example of all three working together:

$ mkproject -t bitbucket virtualenvwrapper.bitbucket
New python executable in virtualenvwrapper.bitbucket/bin/python
Installing distribute.............................................
..................................................................
..................................................................
virtualenvwrapper.user_scripts Creating /Users/dhellmann/.virtualenvs/virtualenvwrapper.bitbucket/bin/predeactivate
virtualenvwrapper.user_scripts Creating /Users/dhellmann/.virtualenvs/virtualenvwrapper.bitbucket/bin/postdeactivate
virtualenvwrapper.user_scripts Creating /Users/dhellmann/.virtualenvs/virtualenvwrapper.bitbucket/bin/preactivate
virtualenvwrapper.user_scripts Creating /Users/dhellmann/.virtualenvs/virtualenvwrapper.bitbucket/bin/postactivate

Creating /Users/dhellmann/Devel/virtualenvwrapper.bitbucket
Applying template bitbucket
virtualenvwrapper.bitbucket Cloning ssh://hg@bitbucket.org/dhellmann/virtualenvwrapper.bitbucket
requesting all changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 10 changes to 8 files
updating to branch default
8 files updated, 0 files merged, 0 files removed, 0 files unresolved
(virtualenvwrapper.bitbucket) $ ls
src
(virtualenvwrapper.bitbucket) $ cat src/.hg/hgrc
[paths]
default = ssh://hg@bitbucket.org/dhellmann/virtualenvwrapper.bitbucket

virtualenvwrapper 2.1 also includes several smaller changes and bug
fixes
(including the pernicious TMPDIR problem several users have
reported).

major new release of virtualenvwrapper

I’m pleased to announce the release of virtualenvwrapper 2.0. This
new version uses a significantly rewritten version of the hook/callback
subsystem to make it easier to share plugins for enhancing developer
workflow. For example, released at the same time is
virtualenvwrapper-emacs-desktop, a plugin to switch emacs project
files when you switch virtualenvs.

I also took this opportunity to change the name of the shell script
containing most of the virtualenvwrapper functionality from
virtualenvwrapper_bashrc to virtualenvwrapper.sh. This
reflects the fact that several shells other than bash are supported
(bash, sh, ksh, and zsh are all reported to
work). You’ll want to update your shell startup file after upgrading
to 2.0.

The work to create the plugin system was triggered by a couple of
recent feature requests for environment templates and for a new
command to create a sub-shell
instead of simply changing the settings
of the current shell. The new, more powerful, plugin capabilities will
make it easier to develop these and similar features.

I’m looking forward to seeing what the community comes up with. I
especially want someone to write a plugin to start a copy of a
development server for a Django project if one is found in a virtualenv.
You’ll get bonus points if it opens the home page of the server in a web
browser.

Updated: Existing user scripts should continue to work
as-written. Any failures are probably a bug, so please report them on
the bitbucket tracker. Documentation for the new plugin system is
available in the virtualenvwrapper docs here.

virtualenvwrapper tips and tricks

I’ve seen a few people talking about how they use virtualenvwrapper
on twitter, including some links to customizations like these aliases
from Holger Krekel.

If you have some tips of your own, I’d appreciate a link in the
comments here. Once I have a few collected, I’ll pull together some of
the interesting ones and add them to the documentation.

So, how do you use virtualenvwrapper?

Switching development contexts with virtualenvwrapper

Optimizing repetitive operations is one of my obsessions. I can’t
stand following long sequences of steps to make the computer do what I
want. The whole point of computers is to have them do more work for
us, right?

As a developer with several ongoing projects, I frequently find myself
switching contexts as one project becomes blocked (read: I lose
interest) and I want to move on another. Typically that means unloading
a bunch of files from my editor and loading others. Some IDEs, and even
editors such as TextMate, use a “project” file to manage the files that
belong to a project. Some go so far as to use the project file as a hint
for how to build the library or application.

Using a project file typically involves pre-defining the members of a
project, and then managing the project file as another artifact of the
project itself. To switch contexts, you simply close one project file
and all of its related code files, then open another. A good editor will
even remember which files were open when the project was last in use,
and restore your workspace to the same state. A great editor will let
you automate the steps needed to switch contexts by loading different
project files.

emacs desktop-mode

I’ve been an emacs user for something like 10 years now. I use vi
frequently, and TextMate regularly as well, but most of the time I live
in emacs for text editing and development tasks. The implementation of
project files I use with emacs is desktop-mode. I like desktop-mode
because instead of forcing me to pre-declare all of the files in my
project, it assumes that all open buffers should be remembered. There
are alternatives (there are always alternatives in emacs, right?).

Setup of desktop-mode is straightforward:

  1. Run customize-group on the “desktop” group.
  2. Turn desktop-save-mode on to enable the minor mode.

3. Optionally change the base name for desktop files in
desktop-base-file-name. I like using “emacs.desktop” so the file is
not hidden and I know exactly what it contains.

4. Set a default search path for the desktop file in desktop-path.
I set my path to include ~/emacs, but this is only the default. We
will be saving a separate desktop in each virtualenv.

5. Set desktop-save to “Always save”. There are other values that
work, but some require interaction with the editor during the context
move to confirm file saves, and I want to avoid that.

There are a few other options that may be useful to tweak, depending
on the other features of emacs you use. For example, I set
desktop-clear-preserve-buffers so clearing the desktop does not delete
the *Messages*, *Org Agenda*, or *scratch* buffers since
those are related to emacs operation and not limited to any one project.

virtualenvwrapper hooks

The next step is to set up the tool chain so emacs loads a new desktop
when you change virtualenvs from the command line.

In addition to making it simpler to manage multiple virtual
environments, virtualenvwrapper adds several hook points to
trigger changes outside of the virtual environment when various events
occur. For example, each time you run workon to change environments,
the predeactivate and postactivate hooks are run. The most
interesting of these for our purposes is the postactivate hook,
called right after the new environment is activated.

I usually add a cd command to the environment-specific postactivate
script (in $VIRTUAL_ENV/bin/postactivate) to move my shell session
to the working directory for that project and change the $PATH,
making it easier to run the tests for my project from the shell. For
example:

pymotw_root=/Users/dhellmann/Documents/PyMOTW
cd $pymotw_root/src
PATH=$pymotw_root/src/bin:$VIRTUAL_ENV/gettext/bin:$PATH

Changing Desktops

In addition to the virtualenv-specific hook, there is also a global
postactivate script, and that’s the one I use to switch contexts in
emacs. Edit $WORKON_HOME/postactivate to add these lines:

# Change emacs' desktop directory
emacsclient -e "(desktop-change-dir "$VIRTUAL_ENV")" >/dev/null

The function desktop-change-dir tells emacs where to save the
desktop session file. By changing the directory each time we change
environments, we can have a separate desktop file in each environment.
Changing the directory automatically saves the old desktop first, of
course, so the old project status is preserved until you come back to
it.

Now that the hook is configured, the next step is to initialize the
desktop for your projects. For each project, run workon to activate
it, then open several files from that project. As you workon the
next project, the session will be saved in
$VIRTUAL_ENV/emacs.desktop (assuming you set your
desktop-base-file-name to emacs.desktop). Once you have initialized a
few virtualenvs, switch back to the first. Emacs should load the desktop
file for that environment and restore the buffers you were editing.

Other Editors

The example above works for emacs, but other editors may need to use
the predeactivate hook to save a project file before loading the new
one from postactivate. There are global and local versions of both
hooks; refer to the virtualenvwrapper documentation for more details.

virtualenvwrapper updates

The latest version available for download is 1.12. New features
include pre and post operation hooks for operations involving
virtualenvs (so you can do things like install your favorite common
packages every time you make a new virtualenv), navigation functions for
working inside the environment, and a host of fixes and optimizations.
Check the release log on PyPI for a complete history of the updates.

In the ~2 weeks since I’ve uploaded virtualenvwrapper to bitbucket,
I’ve received more patches than in all the previous time I maintained
the script. Score 1 for DVCS!

[Updated: Changed “bitkeeper” to “bitbucket” – Sorry Jesper!]