IPython and virtualenv
IPython is a feature-rich interactive shell for Python developers. Virtualenv creates isolated development environments so you can test or install packages without introducing conflicts. This month, Doug examines how both tools can make your life a little easier.
Last month, around the time I started working on my January column, I posted to my blog asking readers to tell me about their favorite Python development tools. I received several good tips from a variety of developers. While some of the responses were scattered all over the map, there were two tools that stood out from the rest with their popularity. The overwhelming favorite of all commenters was IPython, an alternate interactive shell. I had heard about IPython previously from co-workers, but never really looked at it very closely. After the responses to my post, however, I decided I needed to give it a more serious review. The other tool mentioned several times was virtualenv, a “Virtual Python Environment builder”. I was already a virtualenv user, so I was pleased to see it mentioned. Let’s look at virtualenv first, and then use it to test IPython.
virtualenv
Ian Bicking’s virtualenv creates a “private” Python environment, or
sandbox, complete with your own interpreter and site-packages
directory. Having a private sandbox like this is useful in situations
where you don’t have root access to install packages into the global
site-packages area, or where you want to test newer versions of
modules without corrupting your development system.
When you run virtualenv, it sets up a fresh sandbox version of Python
in a directory you specify by copying or linking files from your
default installation to create new bin and lib directories. The
sys.path for the new environment is configured based on options you
provide when you create it. By default, the original Python
environment is included at the end of the search path. This
configuration allows you to share common modules in a global
site-packages directory, so they can be used in several
projects. Listing 1 shows some basic details from a default virtualenv
environment. As you can see starting on line 31, the sys.path has
the user library directories before the global versions of those same
directories and the local lib/python2.5 comes before the global
lib/python2.5, etc.
Listing 1
$ pwd
/Users/dhellmann/Documents/PythonMagazine/tmp
$ virtualenv default
New python executable in default/bin/python
Installing setuptools....................done.
$ find default -type d
default
default/bin
default/lib
default/lib/python2.5
default/lib/python2.5/distutils
default/lib/python2.5/site-packages
$ ls -l default/bin
total 96
-rw-r--r-- 1 dhellmann 501 1213 Jan 31 09:30 activate
-rwxr-xr-x 1 dhellmann 501 329 Jan 31 09:30 easy_install
-rwxr-xr-x 1 dhellmann 501 337 Jan 31 09:30 easy_install-2.5
-rwxr-xr-x 1 dhellmann 501 30028 Jan 31 09:30 python
lrwxr-xr-x 1 dhellmann 501 6 Jan 31 09:30 python2.5 -> python
$ default/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/default/bin/..'
>>> pprint.pprint(sys.path)
['',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/virtualenv-0.9.2-py2.5.egg',
'/Users/dhellmann/PythonMagazine/default/lib/python25.zip',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-darwin',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/plat-mac/lib-scriptpackages',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-tk',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/lib-dynload',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
'/Users/dhellmann/PythonMagazine/default/lib/python2.5/site-packages',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Py2App',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/PyObjC',
]
Alternately, if you specify --no-site-packages when creating the new
environment, you end up with a completely stand-alone environment with
no reference back to the global site-packages. Listing 2 shows the
search path for a private environment without the global
site-packages library directories. Notice that although the global
standard library directory is still included, third-party libraries
are supposed to go into site-packages, so that should not cause any
confusion or conflict when importing modules.
Listing 2
$ virtualenv --no-site-packages private
New python executable in private/bin/python
Installing setuptools............done.
$ private/bin/python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/dhellmann/PythonMagazine/private/bin/..'
>>> import pprint
>>> pprint.pprint(sys.path)
['',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg',
'/Users/dhellmann/PythonMagazine/private/lib/python25.zip',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-darwin',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/plat-mac/lib-scriptpackages',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-tk',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/lib-dynload',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
'/Users/dhellmann/PythonMagazine/private/lib/python2.5/site-packages']
>>>
In both examples so far, I have run the local version of the
interpreter directly so I could examine the module search path. The
sandbox created by virtualenv also includes a simple script called
activate to set up your current shell to use the files in that
sandbox by default. When you source activate, it sets up several
environment variables in your current shell. For example,
VIRTUAL_ENV is set to point to the root of the sandbox, the
directory you specified when you created it. Your shell PATH
variable is also updated to start with the sandbox’s bin
directory. This makes the sandbox version of the interpreter the
default, so when you run scripts they use the libraries from the
sandbox. Any commands you install with easy_install also end up in
$VIRTUAL_ENV/bin, making it easier to run those programs since you
don’t have to remember to type the full path to the program. As a
reminder that you are using a custom environment, your command prompt
also changes to show the name of the virtual environment sandbox.
$ source default/bin/activate
(default)$ echo $VIRTUAL_ENV
/Users/dhellmann/PythonMagazine/default
(default)$ which python
/Users/dhellmann/PythonMagazine/default/bin/python
When I started as Technical Editor here at Python Magazine, I had some concerns about installing all of the dependencies I would need to have in order to test code associated with articles submitted for review by our authors. I knew some articles would have conflicting requirements, and managing those packages was going to be a bit of a hassle. Originally, I planned to set up a virtual machine so I could at least isolate code for the magazine from my own development environment, and wiping it clean between issues (or even articles) would be straightforward. Using virtualenv has turned out to be so much easier that I haven’t bothered with the VM. I typically create a separate environment for each article in a sandbox, verify the dependencies listed by the author along with their code, then delete the entire sandbox. No muss, no fuss.
IPython
IPython is an enhanced interactive shell, intended to be used as a replacement for the standard interactive Python interpreter. Started by Fernando Pérez as a set of enhancements to the basic interpreter prompt, it has grown to include a host of other powerful features and contributions from several developers. Many of the features of IPython derive from its background in scientific computing, but it can be useful outside of scientific fields for anyone who works with Python as well.
The installation instructions have the usual steps (note the use of
sudo to perform the final step with admin privileges):
$ tar -xvzf ipython-0.7.3.tar.gz
$ cd ipython-0.7.3
$ python setup.py build
$ sudo python setup.py install
To illustrate how I usually review contributions for this column,
though, I decided to use easy_install, even though it wasn’t listed
as an option, and virtualenv. Listing 3 shows how I created the new
sandbox and then installed IPython into it.
Listing 3
$ virtualenv ipython
New python executable in ipython/bin/python
Installing setuptools............done.
$ source ipython/bin/activate
(ipython)$
(ipython)$ cd $VIRTUAL_ENV
(ipython)$ ls
bin lib
(ipython)$ easy_install ipython
Searching for ipython
Reading http://pypi.python.org/simple/ipython/
Reading http://ipython.scipy.org
Reading http://ipython.scipy.org/dist
Best match: ipython 0.8.2
Downloading http://ipython.scipy.org/dist/ipython-0.8.2-py2.5.egg
Processing ipython-0.8.2-py2.5.egg
creating /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Extracting ipython-0.8.2-py2.5.egg to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages
Adding ipython 0.8.2 to easy-install.pth file
Installing ipython script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin
Installing pycolor script to /Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin
Installed /Users/dhellmann/PythonMagazine/PythonEnv/ipython/lib/python2.5/site-packages/ipython-0.8.2-py2.5.egg
Processing dependencies for ipython
Finished processing dependencies for ipython
(ipython)$
(ipython)$ which ipython
/Users/dhellmann/PythonMagazine/PythonEnv/ipython/bin/ipython
Prompt History
Now that IPython is installed, let’s start it up and take it for a
spin. Listing 4 shows the “startup screen”. The first thing to notice,
after the boiler-plate help text, is the different prompt, In [1]:.
If your terminal supports it, the prompt will be in color (mine is
green). The different prompt is the first indication of one of the
powerful features of IPython. As you issue instructions, IPython
tracks every input and output expression used in your interactive
session. You can refer back to them to build new expressions in a
couple of different ways. Each input has a history “number”, similar
to the way many Unix shells refer to your command line history. In the
case of IPython, though, this is extended to allow you to refer back
to the outputs as well, without recomputing them. This feature is
especially useful if you are experimenting with code and creating lots
of objects with small variations in their settings.
Listing 4
(ipython)$ ipython
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
Type "copyright", "credits" or "license" for more information.
IPython 0.8.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]:
There are two syntaxes for using back references. You can use built-in
In and Out variables as lists, and index into them. Or if you only
want the output value, you can use the shorthand underscore notation
(_). The values from In are the strings you input, while the
values from Out are the results of evaluating those expressions. For
example:
In [1]: 5*5
Out[1]: 25
In [2]: In[1]
Out[2]: u'5*5n'
In [3]: _1
Out[3]: 25
In [4]: Out[3]
Out[4]: 25
In [5]: _4 * 5
Out[5]: 125
In [6]:
If you are comfortable working at the interactive prompt in this way,
but want to record what you do for future reference after you close
your session, you can use IPython’s logging feature to write the
session to a file. To activate the log, use the control command
%logstart, as illustrated in Listing 5. The output file is a Python
source file, so it is easy to clean it up and turn it into a “real”
module when you are done experimenting.
Listing 5
In [6]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename : ipython_log.py
Mode : rotate
Output logging : False
Raw input log : False
Timestamping : False
State : active
In [7]: a = 5
In [8]: b = 6
In [9]: c = a * b
In [10]: c
Out[10]: 30
In [11]: d = [ a, b, c]
In [12]: d
Out[12]: [5, 6, 30]
In [13]: %logstop
If we examine the file ipython_log.py (Listing 6), we can see that
everything I typed was logged to the file. The command %logstart is
just one of many magic control commands for giving IPython
instructions about how you want to interact with it.
Listing 6
#log# Automatic Logger file. *** THIS MUST BE THE FIRST LINE ***
#log# DO NOT CHANGE THIS LINE OR THE TWO BELOW
#log# opts = Struct({'__allownew': True, 'logfile': 'ipython_log.py'})
#log# args = []
#log# It is safe to make manual edits below here.
#log#-----------------------------------------------------------------------
5*5
In[1]
_1
Out[3]
_4 * 5
_ip.magic("logstart ")
a = 5
b = 6
c = a * b
c
d = [a, b, c]
d
Shell Commands
In addition to standard Python expressions and magic commands, IPython
supports “shelling out” to run commands in your normal command line
shell. This feature, and the fact that some shell commands are built
into IPython, enables some users to use IPython as their primary
command line shell, instead of a more traditional shell such as
bash. Notice the difference in this example between !pwd and pwd –
the latter results in an output value being saved to Out.
In [1]: cd
/Users/dhellmann
In [2]: !pwd
/Users/dhellmann
In [3]: pwd
Out[3]: '/Users/dhellmann'
IPython is smart enough that many system commands can be run directly
from the command prompt without the ! prefix, and any can be run
with the prefix. The text output of the commands can be captured, just
as with more traditional shells, so the values can be used by your
Python code.
Help Is Never Far Away
With all of the additional features, a new user may feel that the
complexity of IPython can be a bit much to take in at once. As with
any good interactive system, help is available right there in the
shell. The %quickref command prints a quick reference card that
lists many of the magic commands, special operators, and other
features with examples and basic instructions for using them. More
specific details are available when you use the ? operator. By
itself, ? displays a help document for IPython itself. It can be
combined with dynamically created objects to show help text for just
about anything in the system (classes, functions, methods, objects,
etc.). Providing built-in help like this makes it easy to find method
signatures and docstrings when you are in the middle of your work,
without having to shift gears to search through online documentation.
Debugging
IPython makes debugging your scripts easier by letting you run them in
the context of the interactive session, then retaining all of the
script state once it is completed. This makes it easy to go back and
look at objects created by the script as though you had typed all of
the commands into the interactive interpreter directly. For example,
take this small bit of sample code in my_sample.py:
#!/usr/bin/env python
my_list = []
my_list.append('first')
my_list.append('second')
print 'End of sample'
When run through IPython, the output looks like this:
In [1]: %run my_sample.py
End of sample
In [2]: my_list
Out[2]: ['first', 'second']
All of the globals from the script are available to the interactive
session, so I can inspect them to see what their settings are. If the
script raises an exception, the source code where the error originates
is displayed along with information about the exception. Working this
way shortens your development cycle because each time you %run a
script, it is reloaded – you don’t have to restart the interpreter to
reset its state.
Advanced Uses
While IPython does have features that make it well suited as an
interactive shell for any Python programmer, it is also being designed
with more advanced purposes in mind. For example, it is possible to
add new syntax profiles so that it understands your own
domain-specific language, making it a natural choice for an embedded
interpreter. For example, IPython comes with a profile to let it
understand units of measure frequently used in physics
computations. This lets you type mass = 3 kg and have it translated
to an object that knows about the 3 as well as “kilograms” to
represent a mass.
Another advanced feature is the ability to control and interact with GUI programs. Usually, GUIs run an infinite event loop to collect input events, process them, and then update the screen. There are bindings available to allow IPython to control most of the popular GUI toolkits so the interactive prompt is not blocked when the event loop runs. This sort of support is integral for combining IPython with something like matplotlib to generate graphs interactively, a key feature for its scientific audience.
In the highly advanced feature category, IPython includes support for controlling parallel processing systems directly from the shell prompt. It supports running Python code distributed over multiple CPUs and hosts, as well as passing Python objects between the processes. Message passing, task farming, and shared memory parallelism are supported. You can even connect and disconnect remote processes dynamically. This lets you start a job and check in later to see how it is running, or share the running state with someone else on your team.
Conclusion
As I mentioned earlier, before I started this series of columns I had already added virtualenv to my standard toolbox, and I use it on a regular basis. Combined with easy_install, I have found it to be irreplaceable for testing new configurations or working on projects with different dependencies. On the other hand, I’m still learning the many features of IPython. It is certainly nicer in many respects than the standard Python shell, but I’m not used to working from the interactive prompt as much as some other developers are, so it may take some time for me to understand its potential fully. Based on the strength of the recommendations I have received and what I have seen so far, I will continue experimenting with it. In the mean time, check it out yourself and let me know what you think.
Next month I will continue this series by introducing you to more
tools to enhance your programming productivity. If you have a tip to
share, feedback on something I’ve written, or if there is a topic you
would like for me to cover in this column, send a note with the
details to doug dot hellmann at pythonmagazine dot com and let me
know, or add the link to your del.icio.us account with the tag
pymagdifferent. And if you’re going to PyCon in March, look me up at
the conference.