IPython and virtualenv

Originally published in Python Magazine Volume 2 Issue 2 , February,
2008

IPython is a feature-rich interactive shell for Python
developers. Virtualenv creates isolated development environments so
you can test or install packages without introducing conflicts. This
month, Doug examines how both tools can make your life a little
easier.

Last month, around the time I started working on my January column, I
posted to my blog asking readers to tell me about their favorite
Python development tools. I received several good tips from a variety
of developers. While some of the responses were scattered all over the
map, there were two tools that stood out from the rest with their
popularity. The overwhelming favorite of all commenters was
IPython, an alternate interactive shell. I had heard about IPython
previously from co-workers, but never really looked at it very
closely. After the responses to my post, however, I decided I needed
to give it a more serious review. The other tool mentioned several
times was virtualenv, a “Virtual Python Environment builder”. I
was already a virtualenv user, so I was pleased to see it mentioned.
Let’s look at virtualenv first, and then use it to test IPython.

virtualenv

Ian Bicking’s virtualenv creates a “private” Python environment, or
sandbox, complete with your own interpreter and site-packages
directory. Having a private sandbox like this is useful in situations
where you don’t have root access to install packages into the global
site-packages area, or where you want to test newer versions of
modules without corrupting your development system.

When you run virtualenv, it sets up a fresh sandbox version of Python
in a directory you specify by copying or linking files from your
default installation to create new bin and lib
directories. The sys.path for the new environment is configured
based on options you provide when you create it. By default, the
original Python environment is included at the end of the search path.
This configuration allows you to share common modules in a global
site-packages directory, so they can be used in several projects.
Listing 1 shows some basic details from a default virtualenv
environment. As you can see starting on line 31, the sys.path has
the user library directories before the global versions of those same
directories and the local lib/python2.5 comes before the global
lib/python2.5, etc.

Listing 1

$ pwd
/Users/dhellmann/Documents/PythonMagazine/tmp

$ virtualenv default
New python executable in default/bin/python
Installing setuptools....................done.

$ find default -type d
default
default/bin
default/lib
default/lib/python2.5
default/lib/python2.5/distutils
default/lib/python2.5/site-packages

$ ls -l default/bin
total 96
-rw-r--r--  1 dhellmann  501   1213 Jan 31 09:30 activate
-rwxr-xr-x  1 dhellmann  501    329 Jan 31 09:30 easy_install
-rwxr-xr-x  1 dhellmann  501    337 Jan 31 09:30 easy_install-2.5
-rwxr-xr-x  1 dhellmann  501  30028 Jan 31 09:30 python
lrwxr-xr-x  1 dhellmann  501      6 Jan 31 09:30 python2.5 -> python

$ default/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/default/bin/..'
>>> pprint.pprint(sys.path)
['',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/virtualenv-0.9.2-py2.5.egg',
 '/Users/dhellmann/PythonMagazine/default/lib/python25.zip',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-darwin',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Py2App',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PyObjC',
]

Alternately, if you specify –no-site-packages when creating the
new environment, you end up with a completely stand-alone environment
with no reference back to the global site-packages. Listing 2
shows the search path for a private environment without the global
site-packages library directories. Notice that although the global
standard library directory is still included, third-party libraries
are supposed to go into site-packages, so that should not cause
any confusion or conflict when importing modules.

Listing 2

$ virtualenv --no-site-packages private
New python executable in private/bin/python
Installing setuptools............done.

$ private/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/private/bin/..'
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
 '/Users/dhellmann/PythonMagazine/private/lib/python25.zip',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-darwin',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages']
>>>

In both examples so far, I have run the local version of the
interpreter directly so I could examine the module search path. The
sandbox created by virtualenv also includes a simple script called
activate to set up your current shell to use the files in that
sandbox by default. When you source activate, it sets up several
environment variables in your current shell. For example,
VIRTUAL_ENV is set to point to the root of the sandbox, the
directory you specified when you created it. Your shell PATH
variable is also updated to start with the sandbox’s bin
directory. This makes the sandbox version of the interpreter the
default, so when you run scripts they use the libraries from the
sandbox. Any commands you install with easy_install also end up in
$VIRTUAL_ENV/bin, making it easier to run those programs since you
don’t have to remember to type the full path to the program. As a
reminder that you are using a custom environment, your command prompt
also changes to show the name of the virtual environment sandbox.

$ source default/bin/activate
(default)$ echo $VIRTUAL_ENV
/Users/dhellmann/PythonMagazine/default
(default)$ which python
/Users/dhellmann/PythonMagazine/default/bin/python

When I started as Technical Editor here at Python Magazine, I had
some concerns about installing all of the dependencies I would need to
have in order to test code associated with articles submitted for
review by our authors. I knew some articles would have conflicting
requirements, and managing those packages was going to be a bit of a
hassle. Originally, I planned to set up a virtual machine so I could
at least isolate code for the magazine from my own development
environment, and wiping it clean between issues (or even articles)
would be straightforward. Using virtualenv has turned out to be so
much easier that I haven’t bothered with the VM. I typically create a
separate environment for each article in a sandbox, verify the
dependencies listed by the author along with their code, then delete
the entire sandbox. No muss, no fuss.

IPython

IPython is an enhanced interactive shell, intended to be used as a
replacement for the standard interactive Python interpreter. Started
by Fernando Pérez as a set of enhancements to the basic interpreter
prompt, it has grown to include a host of other powerful features and
contributions from several developers. Many of the features of IPython
derive from its background in scientific computing, but it can be
useful outside of scientific fields for anyone who works with Python
as well.

The installation instructions have the usual steps (note the use of
sudo to perform the final step with admin privileges):

$ tar -xvzf ipython-0.7.3.tar.gz
$ cd ipython-0.7.3
$ python setup.py build
$ sudo python setup.py install

To illustrate how I usually review contributions for this column,
though, I decided to use easy_install, even though it wasn’t
listed as an option, and virtualenv. Listing 3 shows how I created the
new sandbox and then installed IPython into it.

Listing 3

$ virtualenv ipython
New python executable in ipython/bin/python
Installing setuptools............done.
$ source ipython/bin/activate
(ipython)$

(ipython)$ cd $VIRTUAL_ENV
(ipython)$ ls
bin     lib

(ipython)$ easy_install ipython
Searching for ipython
Reading http://pypi.python.org/simple/ipython/
Reading http://ipython.scipy.org
Reading http://ipython.scipy.org/dist
Best match: ipython 0.8.2
Downloading http://ipython.scipy.org/dist/ipython-0.8.2-py2.5.egg
Processing ipython-0.8.2-py2.5.egg
creating /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Extracting ipython-0.8.2-py2.5.egg to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages
Adding ipython 0.8.2 to easy-install.pth file
Installing ipython script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin
Installing pycolor script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin

Installed /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Processing dependencies for ipython
Finished processing dependencies for ipython
(ipython)$
(ipython)$ which ipython
/Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin/ipython

Prompt History

Now that IPython is installed, let’s start it up and take it for a
spin. Listing 4 shows the “startup screen”. The first thing to notice,
after the boiler-plate help text, is the different prompt, In
[1]:
. If your terminal supports it, the prompt will be in color
(mine is green). The different prompt is the first indication of one
of the powerful features of IPython. As you issue instructions,
IPython tracks every input and output expression used in your
interactive session. You can refer back to them to build new
expressions in a couple of different ways. Each input has a history
“number”, similar to the way many Unix shells refer to your command
line history. In the case of IPython, though, this is extended to
allow you to refer back to the outputs as well, without recomputing
them. This feature is especially useful if you are experimenting with
code and creating lots of objects with small variations in their
settings.

Listing 4

(ipython)$ ipython
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
Type "copyright", "credits" or "license" for more information.

IPython 0.8.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]:

There are two syntaxes for using back references. You can use built-in
In and Out variables as lists, and index into them. Or if you
only want the output value, you can use the shorthand underscore
notation (_). The values from In are the strings you input,
while the values from Out are the results of evaluating those
expressions. For example:

In [1]: 5*5

Out[1]: 25

In [2]: In[1]

Out[2]: u'5*5n'

In [3]: _1

Out[3]: 25

In [4]: Out[3]

Out[4]: 25

In [5]: _4 * 5

Out[5]: 125

In [6]:

If you are comfortable working at the interactive prompt in this way,
but want to record what you do for future reference after you close
your session, you can use IPython’s logging feature to write the
session to a file. To activate the log, use the control command
%logstart, as illustrated in Listing 5. The output file is a
Python source file, so it is easy to clean it up and turn it into a
“real” module when you are done experimenting.

Listing 5

In [6]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

In [7]: a = 5

In [8]: b = 6

In [9]: c = a * b

In [10]: c

Out[10]: 30

In [11]: d = [ a, b, c]

In [12]: d

Out[12]: [5, 6, 30]

In [13]: %logstop

If we examine the file ipython_log.py (Listing 6), we can see that
everything I typed was logged to the file. The command %logstart
is just one of many magic control commands for giving IPython
instructions about how you want to interact with it.

Listing 6

#log# Automatic Logger file. *** THIS MUST BE THE FIRST LINE ***
#log# DO NOT CHANGE THIS LINE OR THE TWO BELOW
#log# opts = Struct({'__allownew': True, 'logfile': 'ipython_log.py'})
#log# args = []
#log# It is safe to make manual edits below here.
#log#-----------------------------------------------------------------------
5*5
In[1]
_1
Out[3]
_4 * 5
_ip.magic("logstart ")

a = 5
b = 6
c = a * b
c
d = [a, b, c]
d

Shell Commands

In addition to standard Python expressions and magic commands, IPython
supports “shelling out” to run commands in your normal command line
shell. This feature, and the fact that some shell commands are built
into IPython, enables some users to use IPython as their primary
command line shell, instead of a more traditional shell such as bash.
Notice the difference in this example between !pwd and pwd
the latter results in an output value being saved to Out.

In [1]: cd
/Users/dhellmann

In [2]: !pwd
/Users/dhellmann

In [3]: pwd

Out[3]: '/Users/dhellmann'

IPython is smart enough that many system commands can be run directly
from the command prompt without the ! prefix, and any can be run
with the prefix. The text output of the commands can be captured, just
as with more traditional shells, so the values can be used by your
Python code.

Help Is Never Far Away

With all of the additional features, a new user may feel that the
complexity of IPython can be a bit much to take in at once. As with
any good interactive system, help is available right there in the
shell. The %quickref command prints a quick reference card that
lists many of the magic commands, special operators, and other
features with examples and basic instructions for using them. More
specific details are available when you use the ? operator. By
itself, ? displays a help document for IPython itself. It can be
combined with dynamically created objects to show help text for just
about anything in the system (classes, functions, methods, objects,
etc.). Providing built-in help like this makes it easy to find method
signatures and docstrings when you are in the middle of your work,
without having to shift gears to search through online documentation.

Debugging

IPython makes debugging your scripts easier by letting you run them in
the context of the interactive session, then retaining all of the
script state once it is completed. This makes it easy to go back and
look at objects created by the script as though you had typed all of
the commands into the interactive interpreter directly. For example,
take this small bit of sample code in my_sample.py:

#!/usr/bin/env python
my_list = []
my_list.append('first')
my_list.append('second')
print 'End of sample'

When run through IPython, the output looks like this:

In [1]: %run my_sample.py
End of sample

In [2]: my_list

Out[2]: ['first', 'second']

All of the globals from the script are available to the interactive
session, so I can inspect them to see what their settings are. If the
script raises an exception, the source code where the error originates
is displayed along with information about the exception. Working this
way shortens your development cycle because each time you %run a
script, it is reloaded – you don’t have to restart the interpreter to
reset its state.

Advanced Uses

While IPython does have features that make it well suited as an
interactive shell for any Python programmer, it is also being designed
with more advanced purposes in mind. For example, it is possible to
add new syntax profiles so that it understands your own
domain-specific language, making it a natural choice for an embedded
interpreter. For example, IPython comes with a profile to let it
understand units of measure frequently used in physics computations.
This lets you type mass = 3 kg and have it translated to an object
that knows about the 3 as well as “kilograms” to represent a mass.

Another advanced feature is the ability to control and interact with
GUI programs. Usually, GUIs run an infinite event loop to collect
input events, process them, and then update the screen. There are
bindings available to allow IPython to control most of the popular GUI
toolkits so the interactive prompt is not blocked when the event loop
runs. This sort of support is integral for combining IPython with
something like matplotlib to generate graphs interactively, a key
feature for its scientific audience.

In the highly advanced feature category, IPython includes support for
controlling parallel processing systems directly from the shell
prompt. It supports running Python code distributed over multiple CPUs
and hosts, as well as passing Python objects between the
processes. Message passing, task farming, and shared memory
parallelism are supported. You can even connect and disconnect remote
processes dynamically. This lets you start a job and check in later to
see how it is running, or share the running state with someone else on
your team.

Conclusion

As I mentioned earlier, before I started this series of columns I had
already added virtualenv to my standard toolbox, and I use it on a
regular basis. Combined with easy_install, I have found it to be
irreplaceable for testing new configurations or working on projects
with different dependencies. On the other hand, I’m still learning the
many features of IPython. It is certainly nicer in many respects than
the standard Python shell, but I’m not used to working from the
interactive prompt as much as some other developers are, so it may
take some time for me to understand its potential fully. Based on the
strength of the recommendations I have received and what I have seen
so far, I will continue experimenting with it. In the mean time, check
it out yourself and let me know what you think.

Next month I will continue this series by introducing you to more
tools to enhance your programming productivity. If you have a tip to
share, feedback on something I’ve written, or if there is a topic you
would like for me to cover in this column, send a note with the
details to doug dot hellmann at pythonmagazine dot com and let me
know, or add the link to your del.icio.us account with the tag
pymagdifferent. And if you’re going to PyCon in March, look me up
at the conference.