PyMOTW: traceback

The traceback module works with the call stack to produce error
messages. A traceback is a stack trace from the point of an exception
handler down the call chain to the point where the exception was
raised. You can also work with the current call stack up from the
point of a call (and without the context of an error), which is useful
for finding out the paths being followed into a function.

Read more at pymotw.com: traceback

PyMOTW: heapq

A heap is a tree-like data structure where the child nodes have a
sort-order relationship with the parents. Binary heaps can be
represented using a list or array organized so that the children of
element N are at positions 2*N+1 and 2*N+2 (for zero-based indexes).
This feature makes it possible to rearrange heaps in place, so it is
not necessary to reallocate as much memory when adding or removing
items.

Read more at pymotw.com: heapq

PyMOTW: cmd

The cmd module contains one public class, Cmd, designed to be
used as a base class for command processors such as interactive shells
and other command interpreters. By default it uses readline for
interactive prompt handling, command line editing, and command
completion.

Read more at pymotw.com: cmd

virtualenvwrapper

Not content to leave well enough alone, Doug offers up some
extensions to Ian Bicking’s virtualenv script that make it even
more useful.

Back in February, I introduced Ian Bicking’s virtualenv script for creating
isolated Python environments, complete with their own interpreter and
site-packages directory. I’ve been using virtualenv for all of my
projects since then, but keeping up with all of the environments I
have been creating has turned a little messy. I’ve solved the problem
by writing a few wrappers in the
form of bash functions. If you are a virtualenv user, you may find
them useful, too.

virtualenv

As I mentioned in the previous column, bringing you interesting
articles about new projects from the far reaches of the internet means
that I end up installing a lot of Python packages to review code
before it is published. That’s okay, but I want my system to be
usable for working on my own development projects, too. Of course,
creating a separate “sandbox” for each article or project means I can
install whatever libraries I need, and not worry about version
conflicts or deleting something accidentally. It’s almost like Ian
wrote virtualenv specifically for me.

For new subscribers who missed the February column, here’s a quick
primer in virtualenv: Every time you run virtualenv, it sets up a
clean copy of Python in a new directory by copying or linking files
from your primary Python installation to create new bin and
lib directories. To use the sandbox, you simply run source
$path/bin/activate
. Your environment is then reconfigured so that
the version of Python in the virtual environment takes precedent over
the normal version installed globally on the system. Since you
created the directory yourself, you have all the permissions you need
to install new modules or libraries into the environment. And since
the environments are light weight and easy to create, you can have as
many of them as you want.

Too Many Virtual Environments

So far, so good. An average user would probably create a new sandbox
for each ongoing project, with some temporary environments thrown in
once in a while for testing a new version of a library or playing with
something discovered through the Python Package Index feed. I, on the
other hand, found myself creating 4-6 new environments every few weeks
as I created a separate environment for each article being reviewed
for the magazine. The number of environments I had quickly grew to be
unmanageable.

I started with all of my virtual environments in the same directories
that held the other files for the article, so I could easily find the
activate script used to enable the environment. This way of
working meant I had environments scattered all through out my
Documents folder, though, and I had to cd to each directory to
make sure I was looking at the right environment before activating it.
Another complication was the temporary environments were mixed
together with “real” files that made up the articles and other parts
of the magazine. Since we use version control to manage those files,
I had to constantly tell the version control tools to ignore the
virtual environment files.

I decided the first change I needed to make was to just put all of the
environments together in one place, so I could see them all easily and
select the right one quickly. I created a directory,
~/.virtualenvs to hold all of my new virtual environments, and
moved my existing environments there. I gave each one a name based on
the project or article it was associated with to make it easy to tell
them apart.

To create a new environment, all I had to do was:

$ cd ~/.virtualenvs
$ virtualenv newname
$ cd -

which I eventually shortened to just:

$ (cd ~/.virtualenvs; virtualenv newname)

I type pretty quickly, but I’m still averse to repeating myself. The
only part of the command that changed each time I made a new
environment was the newname argument to virtualenv. It looked
like a perfect opportunity to use a wrapper to streamline the command.
I thought about using a shell alias, but my bash alias-fu isn’t all
that strong and I wanted to include some basic error checking.
Creating a shell function however, was very easy, and combined all
of the benefits of an alias with the syntax of a script.

As these “simple” projects tend to do, this one quickly grew to
include a few more features. Eventually it reached the point where my
desire to hack on it was satisfied and I could actually use it. The
results, called virtualenvwrapper, are available in Listing 1.

Managing Environments

The bash script in Listing 1 is intended to be run during your shell
login sequence via the source command. The distributed version of
the file is (unimaginatively) named virtualenvwrapper_bashrc, so
to use it you would add a line like this to the end of your
~/.bashrc file:

source $HOME/bin/virtualenvwrapper_bashrc

Of course the actual path depends on where you put the file when you
download it. There is just the one file, so I keep a copy in
$HOME/bin, but you might have other standard practices for
extension scripts like this one.

The only other setup steps you need to take before using the wrapper
is to define the shell variable WORKON_HOME to point to the
directory where your virtual environments will go. The default value
is ~/.virtualenvs, and you don’t have to change it if you want to
put them somewhere else. You can place the setting before or after
you source the script, so you would end up with something like this:

export WORKON_HOME="$HOME/Devel/Environments"
source $HOME/bin/virtualenvwrapper_bashrc

Once you have both commands in your login script, just source
~/.bashrc
and then you can start creating
environments. mkvirtualenv is a very thin wrapper around
virtualenv itself that creates an environment and then immediately
activates it.

$ mkvirtualenv newenv
New python executable in newenv/bin/python
Installing setuptools....................done.
(newenv)farnsworth:dhellmann:~:502 $

In fact, all of the arguments you give to mkvirtualenv are passed
directly to virtualenv, so you can use -h, –no-site-packages,
and all of the other normal options.

Switching Environments

One of the primary reasons I created virtualenvwrapper was to make it
easier for me to alternate between different environments from the
same shell. The workon function is the interface for switching,
named that way because I use it when I want to “work on” a different
project. To see a list of available environments, run workon with
no arguments. To switch to an environment, give the name as an
argument.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
(newenv)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
newenv
personal
pymag
pymotw

As you see here, most of the available environments are for my own
personal projects. You may recognize PyGameTutorial as being the
environment I have been using to test Terry Hancock’s code for his
tutorial series, the first installment of which appears elsewhere in
this issue. To switch to that environment, I simply run workon
PyGameTutorial
:

(newenv)$ workon PyGameTutorial
(PyGameTutorial)$

To switch to the sandbox I use for writing the “Python Module of the
Week”, I would run:

(PyGameTutorial)$ workon pymotw
(pymotw)$

Fancy Switching

Sometimes, as with the PyMOTW series, I want to change more about the
shell environment than just the virtual environment settings. For
example, I have a working directory where I create all of the code for
PyMOTW, and when I “switch” to working on it, I want my shell to go
there. To allow those sorts of actions, I added a feature to
workon to look for hook scripts inside the $VIRTUAL_ENV/bin
directory, and run them before and after activating the environment.

Before you move out of an environment, you usually want to save the
project in your editor, clean up temporary files, and otherwise
preserve its current state. The predeactivate hook offers an
opportunity to clean up the current environment, before switching to
the new one.

If the postactivate script is present, it is run after the new
environment is configured. So to do any extra customization for an
environment, you can simply create the postactivate script with
the commands and they will be run automatically. You might want to
open a new project file in your editor, set your terminal window
title, or take any number of other customization steps.

Both of the hook scripts are sourced in the current shell, rather than
being run as a new program, to give them an opportunity to change
active environment variables. You can use them to change your
PATH or re-configure your shell prompt, for example. Anything you
can do from the shell command line can be done in the pre and
post activation scripts.

In the case of PyMOTW, I have a simple call to cd in the
postactivate script. I include a similar command in the
postactivate script for each temporary environment I create for a
new article, to make it easy to move around within the magazine source
directories.

Cleaning Up

The problem that led me to create these shell functions in the first
place was that I had too many virtual environments floating around my
hard drive. At this point I have described tools I wrote to make
creating new environments even easier. I also added a simple
function, rmvirtualenv to make removing environments safe and
easy.

Removing an environment is simple with rm, but after accidentally
removing my working environment once, I thought the extra check was
worth a new command. rmvirtualenv lets you remove any environment
by name, as long as it is not the currently activated environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
(pymotw)$ rmvirtualenv pymotw
ERROR: You cannot remove the active environment.
(pymotw)$ rmvirtualenv newenv
(pymotw)$ workon
2.5
CastSampler
PyGameTutorial
docket
loghetti
personal
pymag
pymotw

Conclusions

Some readers may wonder why I gave the shell functions
mkvirtualenv and rmvirtualenv such long names if I was trying
to avoid doing so much typing. Using tab completion at the shell
prompt, I never have to type the whole command. mkvirt or
rmvirt followed by TAB expands to the complete command name.

Although virtualenvwrapper meets my needs today, there are a few more
features I would like to add, eventually. No software is ever
actually done, right?

rmvirtualenv really needs another safety switch to make sure the
user means it. It could, for example, list all of the modules or eggs
installed in the private site-packages directory and then prompt
the user to make sure they really want to remove the environment.

workon should offer a “none” option, to clear the virtual
environment settings from the current shell without switching to
another one. For now I work around that by having an empty “2.5”
environment that mirrors my installed site-packages.

As I have said before, virtualenv has become an integral part of my
toolbox. Whether for my own projects or editorial reviews, the
ability to create a clean scratch area where I can install software
with impunity gives me a sense of freedom that working directly out of
/usr/local just can’t match. I hope you give virtualenv a try,
and if it suits you, maybe virtualenvwrapper will be a useful addition
as well.

Next Month

Next month this series will continue with coverage of more tools to
enhance your programming productivity. If you have a tip to share,
feedback on something I’ve written, or if there is a topic you would
like for me to cover in this column, send a note with the details to
doug dot hellmann at pythonmagazine dot com and let me know, or add
the link to your del.icio.us account with the tag pymagdifferent.

Originally published in Python Magazine Volume 2 Issue 5 , May, 2008

PyMOTW: functools

The functools module includes tools for wrapping functions and
other callable objects.

The primary tool supplied by the functools module is the class
partial, which can be used to “wrap” a callable with default
arguments. The resulting object is itself callable and can be treated
as though it is the original function. It takes all of the same
arguments as the original callable and can be invoked with extra
positional or named arguments as well.

Read more at pymotw.com: functools

PyMOTW: operator

Functional programming using iterators occasionally requires you to
create small functions for simple expressions. Sometimes these can be
expressed as lambda functions. But for some operations, you don’t
need to define your own function at all. The operator module defines
functions that correspond to built-in operations for arithmetic, and
comparison as well as sequence and dictionary operations.

Read more at pymotw.com: operator

One year of “The Python Module of the Week”

It’s a bit passé to recognize blogging anniversaries, but as it’s my
first I’m going to do a little retrospecting anyway. :-)

I just realized this afternoon that I had missed celebrating the first
anniversary of PyMOTW by a few weeks. I started the series as an excuse
to force myself to write something once a week. At the time, it seemed
like a somewhat lame idea and I wasn’t sure I would keep it up. There
are any number of reference guides for the standard library out there.
Sitting down to read through one isn’t that exciting, though, so I
thought writing example code with all of the modules would be a way to
force myself to actually study the modules I didn’t use on a regular
basis.

The first real post from 25 March 2007 covered the fileinput
module. It wasn’t until several posts into the series that I started
collecting and releasing the code through PyPI, so the version number
for the source package is only up to 1.48 even though I’ve done more
than 52 weeks worth of modules. (The os module took 4 weeks, so I
haven’t done as many modules as weeks of posts.)

My Writing Process

My process for creating the posts has changed substantially over the
last year. The first few posts were posted through the web form on
blogger.com. They consisted of a lot of hand-edited HTML combined with
output from the web version of pygments (used to highlight the syntax
in the code examples). I also used to write the prose for each post
first, and the code samples later.

Now, I have the entire process reversed. I work through all of the
code examples before writing any prose. The code comes more quickly, and
I can revise and refactor it so the examples work together without
having to go back and edit the rest of the text. Once I have the code
finished, I use a combination of shortcuts I’ve built for TextMate
and MarsEdit to assemble the post and write the prose portions. It
takes me a lot less time to create a single post now that I’ve refined
the workflow. The post on the operator module from today, for example,
only took a couple of hours (with interruptions). It is a little skimpy
on prose, though.

Future Plans

Since the beginning, I’ve had a fair number of comments (online and
off) from people who tell me that the posts have been personally useful
to them. I appreciate that sort of feedback, and it motivates me to keep
going. I’m running out of the “simple” modules, and as I’ve also started
working on Python Magazine over the past year, I don’t actually have the
same amount of free time any more. Having a bit of extra motivation will
spur me to pick up some of the bigger modules like email and
elementtree.

At the rate I’m going, I’m not going to finish the whole library
before Python 3.0 comes out, and the current plans call for some
modules to be removed, deprecated APIs to be dropped, and other sorts of
changes. The rules say some modules can even be renamed. When that
settles down and there is an actual release, I’ll probably stop writing
about 2.x and pick up with 3.0. I haven’t decided yet, though.

Python Module of the Week Home