IPython and virtualenv

Originally published in Python Magazine Volume 2 Issue 2 , February, 2008

IPython is a feature-rich interactive shell for Python developers. Virtualenv creates isolated development environments so you can test or install packages without introducing conflicts. This month, Doug examines how both tools can make your life a little easier.

Last month, around the time I started working on my January column, I posted to my blog asking readers to tell me about their favorite Python development tools. I received several good tips from a variety of developers. While some of the responses were scattered all over the map, there were two tools that stood out from the rest with their popularity. The overwhelming favorite of all commenters was IPython, an alternate interactive shell. I had heard about IPython previously from co-workers, but never really looked at it very closely. After the responses to my post, however, I decided I needed to give it a more serious review. The other tool mentioned several times was virtualenv, a “Virtual Python Environment builder”. I was already a virtualenv user, so I was pleased to see it mentioned. Let’s look at virtualenv first, and then use it to test IPython.

virtualenv

Ian Bicking’s virtualenv creates a “private” Python environment, or sandbox, complete with your own interpreter and site-packages directory. Having a private sandbox like this is useful in situations where you don’t have root access to install packages into the global site-packages area, or where you want to test newer versions of modules without corrupting your development system.

When you run virtualenv, it sets up a fresh sandbox version of Python in a directory you specify by copying or linking files from your default installation to create new bin and lib directories. The sys.path for the new environment is configured based on options you provide when you create it. By default, the original Python environment is included at the end of the search path. This configuration allows you to share common modules in a global site-packages directory, so they can be used in several projects. Listing 1 shows some basic details from a default virtualenv environment. As you can see starting on line 31, the sys.path has the user library directories before the global versions of those same directories and the local lib/python2.5 comes before the global lib/python2.5, etc.

Listing 1

$ pwd
/Users/dhellmann/Documents/PythonMagazine/tmp

$ virtualenv default
New python executable in default/bin/python
Installing setuptools....................done.

$ find default -type d
default
default/bin
default/lib
default/lib/python2.5
default/lib/python2.5/distutils
default/lib/python2.5/site-packages

$ ls -l default/bin
total 96
-rw-r--r--  1 dhellmann  501   1213 Jan 31 09:30 activate
-rwxr-xr-x  1 dhellmann  501    329 Jan 31 09:30 easy_install
-rwxr-xr-x  1 dhellmann  501    337 Jan 31 09:30 easy_install-2.5
-rwxr-xr-x  1 dhellmann  501  30028 Jan 31 09:30 python
lrwxr-xr-x  1 dhellmann  501      6 Jan 31 09:30 python2.5 -> python

$ default/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/default/bin/..'
>>> pprint.pprint(sys.path)
['',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/virtualenv-0.9.2-py2.5.egg',
 '/Users/dhellmann/PythonMagazine/default/lib/python25.zip',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-darwin',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Py2App',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PyObjC',
]

Alternately, if you specify --no-site-packages when creating the new environment, you end up with a completely stand-alone environment with no reference back to the global site-packages. Listing 2 shows the search path for a private environment without the global site-packages library directories. Notice that although the global standard library directory is still included, third-party libraries are supposed to go into site-packages, so that should not cause any confusion or conflict when importing modules.

Listing 2

$ virtualenv --no-site-packages private
New python executable in private/bin/python
Installing setuptools............done.

$ private/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/private/bin/..'
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
 '/Users/dhellmann/PythonMagazine/private/lib/python25.zip',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-darwin',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac/lib-scriptpackages',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
 '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
 '/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages']
>>>

In both examples so far, I have run the local version of the interpreter directly so I could examine the module search path. The sandbox created by virtualenv also includes a simple script called activate to set up your current shell to use the files in that sandbox by default. When you source activate, it sets up several environment variables in your current shell. For example, VIRTUAL_ENV is set to point to the root of the sandbox, the directory you specified when you created it. Your shell PATH variable is also updated to start with the sandbox’s bin directory. This makes the sandbox version of the interpreter the default, so when you run scripts they use the libraries from the sandbox. Any commands you install with easy_install also end up in $VIRTUAL_ENV/bin, making it easier to run those programs since you don’t have to remember to type the full path to the program. As a reminder that you are using a custom environment, your command prompt also changes to show the name of the virtual environment sandbox.

$ source default/bin/activate
(default)$ echo $VIRTUAL_ENV
/Users/dhellmann/PythonMagazine/default
(default)$ which python
/Users/dhellmann/PythonMagazine/default/bin/python

When I started as Technical Editor here at Python Magazine, I had some concerns about installing all of the dependencies I would need to have in order to test code associated with articles submitted for review by our authors. I knew some articles would have conflicting requirements, and managing those packages was going to be a bit of a hassle. Originally, I planned to set up a virtual machine so I could at least isolate code for the magazine from my own development environment, and wiping it clean between issues (or even articles) would be straightforward. Using virtualenv has turned out to be so much easier that I haven’t bothered with the VM. I typically create a separate environment for each article in a sandbox, verify the dependencies listed by the author along with their code, then delete the entire sandbox. No muss, no fuss.

IPython

IPython is an enhanced interactive shell, intended to be used as a replacement for the standard interactive Python interpreter. Started by Fernando Pérez as a set of enhancements to the basic interpreter prompt, it has grown to include a host of other powerful features and contributions from several developers. Many of the features of IPython derive from its background in scientific computing, but it can be useful outside of scientific fields for anyone who works with Python as well.

The installation instructions have the usual steps (note the use of sudo to perform the final step with admin privileges):

$ tar -xvzf ipython-0.7.3.tar.gz
$ cd ipython-0.7.3
$ python setup.py build
$ sudo python setup.py install

To illustrate how I usually review contributions for this column, though, I decided to use easy_install, even though it wasn’t listed as an option, and virtualenv. Listing 3 shows how I created the new sandbox and then installed IPython into it.

Listing 3

$ virtualenv ipython
New python executable in ipython/bin/python
Installing setuptools............done.
$ source ipython/bin/activate
(ipython)$

(ipython)$ cd $VIRTUAL_ENV
(ipython)$ ls
bin     lib

(ipython)$ easy_install ipython
Searching for ipython
Reading http://pypi.python.org/simple/ipython/
Reading http://ipython.scipy.org
Reading http://ipython.scipy.org/dist
Best match: ipython 0.8.2
Downloading http://ipython.scipy.org/dist/ipython-0.8.2-py2.5.egg
Processing ipython-0.8.2-py2.5.egg
creating /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Extracting ipython-0.8.2-py2.5.egg to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages
Adding ipython 0.8.2 to easy-install.pth file
Installing ipython script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin
Installing pycolor script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin

Installed /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Processing dependencies for ipython
Finished processing dependencies for ipython
(ipython)$
(ipython)$ which ipython
/Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin/ipython

Prompt History

Now that IPython is installed, let’s start it up and take it for a spin. Listing 4 shows the “startup screen”. The first thing to notice, after the boiler-plate help text, is the different prompt, In [1]:. If your terminal supports it, the prompt will be in color (mine is green). The different prompt is the first indication of one of the powerful features of IPython. As you issue instructions, IPython tracks every input and output expression used in your interactive session. You can refer back to them to build new expressions in a couple of different ways. Each input has a history “number”, similar to the way many Unix shells refer to your command line history. In the case of IPython, though, this is extended to allow you to refer back to the outputs as well, without recomputing them. This feature is especially useful if you are experimenting with code and creating lots of objects with small variations in their settings.

Listing 4

(ipython)$ ipython
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
Type "copyright", "credits" or "license" for more information.

IPython 0.8.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]:

There are two syntaxes for using back references. You can use built-in In and Out variables as lists, and index into them. Or if you only want the output value, you can use the shorthand underscore notation (_). The values from In are the strings you input, while the values from Out are the results of evaluating those expressions. For example:

In [1]: 5*5

Out[1]: 25

In [2]: In[1]

Out[2]: u'5*5n'

In [3]: _1

Out[3]: 25

In [4]: Out[3]

Out[4]: 25

In [5]: _4 * 5

Out[5]: 125

In [6]:

If you are comfortable working at the interactive prompt in this way, but want to record what you do for future reference after you close your session, you can use IPython’s logging feature to write the session to a file. To activate the log, use the control command %logstart, as illustrated in Listing 5. The output file is a Python source file, so it is easy to clean it up and turn it into a “real” module when you are done experimenting.

Listing 5

In [6]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

In [7]: a = 5

In [8]: b = 6

In [9]: c = a * b

In [10]: c

Out[10]: 30

In [11]: d = [ a, b, c]

In [12]: d

Out[12]: [5, 6, 30]

In [13]: %logstop

If we examine the file ipython_log.py (Listing 6), we can see that everything I typed was logged to the file. The command %logstart is just one of many magic control commands for giving IPython instructions about how you want to interact with it.

Listing 6

#log# Automatic Logger file. *** THIS MUST BE THE FIRST LINE ***
#log# DO NOT CHANGE THIS LINE OR THE TWO BELOW
#log# opts = Struct({'__allownew': True, 'logfile': 'ipython_log.py'})
#log# args = []
#log# It is safe to make manual edits below here.
#log#-----------------------------------------------------------------------
5*5
In[1]
_1
Out[3]
_4 * 5
_ip.magic("logstart ")

a = 5
b = 6
c = a * b
c
d = [a, b, c]
d

Shell Commands

In addition to standard Python expressions and magic commands, IPython supports “shelling out” to run commands in your normal command line shell. This feature, and the fact that some shell commands are built into IPython, enables some users to use IPython as their primary command line shell, instead of a more traditional shell such as bash. Notice the difference in this example between !pwd and pwd – the latter results in an output value being saved to Out.

In [1]: cd
/Users/dhellmann

In [2]: !pwd
/Users/dhellmann

In [3]: pwd

Out[3]: '/Users/dhellmann'

IPython is smart enough that many system commands can be run directly from the command prompt without the ! prefix, and any can be run with the prefix. The text output of the commands can be captured, just as with more traditional shells, so the values can be used by your Python code.

Help Is Never Far Away

With all of the additional features, a new user may feel that the complexity of IPython can be a bit much to take in at once. As with any good interactive system, help is available right there in the shell. The %quickref command prints a quick reference card that lists many of the magic commands, special operators, and other features with examples and basic instructions for using them. More specific details are available when you use the ? operator. By itself, ? displays a help document for IPython itself. It can be combined with dynamically created objects to show help text for just about anything in the system (classes, functions, methods, objects, etc.). Providing built-in help like this makes it easy to find method signatures and docstrings when you are in the middle of your work, without having to shift gears to search through online documentation.

Debugging

IPython makes debugging your scripts easier by letting you run them in the context of the interactive session, then retaining all of the script state once it is completed. This makes it easy to go back and look at objects created by the script as though you had typed all of the commands into the interactive interpreter directly. For example, take this small bit of sample code in my_sample.py:

#!/usr/bin/env python
my_list = []
my_list.append('first')
my_list.append('second')
print 'End of sample'

When run through IPython, the output looks like this:

In [1]: %run my_sample.py
End of sample

In [2]: my_list

Out[2]: ['first', 'second']

All of the globals from the script are available to the interactive session, so I can inspect them to see what their settings are. If the script raises an exception, the source code where the error originates is displayed along with information about the exception. Working this way shortens your development cycle because each time you %run a script, it is reloaded – you don’t have to restart the interpreter to reset its state.

Advanced Uses

While IPython does have features that make it well suited as an interactive shell for any Python programmer, it is also being designed with more advanced purposes in mind. For example, it is possible to add new syntax profiles so that it understands your own domain-specific language, making it a natural choice for an embedded interpreter. For example, IPython comes with a profile to let it understand units of measure frequently used in physics computations. This lets you type mass</span> <span class="pre">=</span> <span class="pre">3</span> <span class="pre">kg and have it translated to an object that knows about the 3 as well as “kilograms” to represent a mass.

Another advanced feature is the ability to control and interact with GUI programs. Usually, GUIs run an infinite event loop to collect input events, process them, and then update the screen. There are bindings available to allow IPython to control most of the popular GUI toolkits so the interactive prompt is not blocked when the event loop runs. This sort of support is integral for combining IPython with something like matplotlib to generate graphs interactively, a key feature for its scientific audience.

In the highly advanced feature category, IPython includes support for controlling parallel processing systems directly from the shell prompt. It supports running Python code distributed over multiple CPUs and hosts, as well as passing Python objects between the processes. Message passing, task farming, and shared memory parallelism are supported. You can even connect and disconnect remote processes dynamically. This lets you start a job and check in later to see how it is running, or share the running state with someone else on your team.

Conclusion

As I mentioned earlier, before I started this series of columns I had already added virtualenv to my standard toolbox, and I use it on a regular basis. Combined with easy_install, I have found it to be irreplaceable for testing new configurations or working on projects with different dependencies. On the other hand, I’m still learning the many features of IPython. It is certainly nicer in many respects than the standard Python shell, but I’m not used to working from the interactive prompt as much as some other developers are, so it may take some time for me to understand its potential fully. Based on the strength of the recommendations I have received and what I have seen so far, I will continue experimenting with it. In the mean time, check it out yourself and let me know what you think.

Next month I will continue this series by introducing you to more tools to enhance your programming productivity. If you have a tip to share, feedback on something I’ve written, or if there is a topic you would like for me to cover in this column, send a note with the details to doug dot hellmann at pythonmagazine dot com and let me know, or add the link to your del.icio.us account with the tag pymagdifferent. And if you’re going to PyCon in March, look me up at the conference.