virtualenvwrapper.django 0.2

What is virtualenvwrapper.django?

virtualenvwrapper.django is a template for virtualenvwrapper to
extend its project-management features. It automatically creates a
new Django site when creating a new project directory.

Refer to
http://www.doughellmann.com/projects/virtualenvwrapper.django/ for
details.

What’s New in 0.2?

This release shifts from requiring virtualenvwrapper.project to using
the new version of virtualenvwrapper that includes the project
management features.

Caching RSS Feeds With feedcache

The past several years have seen a steady increase in the use of RSS
and Atom feeds for data sharing. Blogs, podcasts, social networking
sites, search engines, and news services are just a few examples of data
sources delivered via such feeds. Working with internet services
requires care, because inefficiencies in one client implementation may
cause performance problems with the service that can be felt by all of
the consumers accessing the same server. In this article, I describe the
development of the feedcache package, and give examples of how you can
use it to optimize the use of data feeds in your application.

Read more

Django with PostgreSQL on Mac OS X Leopard

Previously, I discussed the steps I went through to get PostgreSQL
working on Tiger
. This weekend I upgraded my system to a new MacBook
Pro running Leopard.

PPC -> x86

Although the Migration Assistant copied the version of PostgreSQL I
had previously installed to the new machine, the results didn’t work
because the service would not start correctly. I ended up reinstalling
using the Unified Installer from PostgreSQL for Mac, and the server
still wouldn’t start. I deleted the old database and re-initialized it
(thanks to hints in some instructions from Russell Brooks) and that
took care of the problem. I’m not sure if there was any way for me to
convert the data, but I didn’t have anything important in the database
that I can’t re-create, so it’s fine to start from scratch.

X Code

The next step was to install X Code 3. That was easy, with the package
installer from Apple.

psycopg

And then I went back to battle with my old nemesis psycopg. I should
probably have taken Steve Holden’s advice in the comments on my earlier
post and just used psycopg2 instead. I didn’t because that would have
meant upgrading my production server, too, since the whole point of
using PostgreSQL instead of SQLite is the back-end adapters in django
produce different SQL for the same QuerySet.

To configure psycopg, I had to set LDFLAGS to include the directory
with the crt1.10.5.0 library. It’s installed to what looks like
should be a standard library directory for the X Code gcc, but ld
couldn’t find it during the “create an executable” test.

Then when running make I had the same problem I had seen under
PPC:

$ make
gcc  -DNDEBUG -g -O3  -I/Users/dhellmann/Devel/AthensDocket/bin/../include/python2.5
-I/Users/dhellmann/Devel/AthensDocket/bin/../lib/python2.5/config -DPACKAGE_NAME="psycopg"
-DPACKAGE_TARNAME="psycopg" -DPACKAGE_VERSION="1.1.21" -DPACKAGE_STRING="psycopg 1.1.21"
-DPACKAGE_BUGREPORT="psycopg@lists.initd.org" -DHAVE_LIBCRYPTO=1 -DHAVE_ASPRINTF=1
-I/Library/PostgreSQL8/include -I/Library/PostgreSQL8/include/postgresql/server
-I../egenix-mx-base-3.0.0/mx/DateTime/mxDateTime -DHAVE_PQFREENOTIFY -DNDEBUG -D_REENTRANT
-D_GNU_SOURCE -DPOSTGRESQL_MAJOR=8 -DPOSTGRESQL_MINOR=2   -c ././module.c -o ./module.o
In file included from ././module.c:33:
././module.h:30:20: error: Python.h: No such file or directory

I thought this was related to using virtualenv, but it didn’t work
correctly outside of the virtualenv setting this time (for some reason,
it did on the PPC laptop). It turns out the error message was correct
and configure/gcc just couldn’t tell where the Python headers were. The
configure command that let me compile was:

$ CPPFLAGS=-I/Library/Frameworks/Python.framework/Headers 
> LDFLAGS=-L/Developer/SDKs/MacOSX10.5.sdk/usr/lib 
> ./configure --with-postgres-libraries=/Library/PostgreSQL8/lib 
> --with-postgres-includes=/Library/PostgreSQL8/include 
> --with-mxdatetime-includes=../egenix-mx-base-3.0.0/mx/DateTime/mxDateTime

Then, oddly enough, when I did my make install step, the
psycopg.so was copied to $VIRTUAL_ENV/bin instead of
$VIRTUAL_ENV/lib/python2.5/site-packages. That was easy enough to
solve by moving the file manually, and then I was able to import the
psycopg module.

So, after about an hour, I’m back to being able to develop with django
and PostgreSQL on OS X Leopard. Maybe now I can start enjoying some of
the new features!

Using raw SQL in django

For a django project I’m working on, I need to run a simple query with
COUNT and GROUP BY statements to collect some summary data. I
tried working with the model API, and eventually fell back to using raw
SQL instead.

Here’s the query I wanted to run:

SELECT
    (year/10)*10 as decade,
    count(*) as case_count
FROM
    docket_case
GROUP BY
    decade

One problem I encountered was there was no way (that I could find) to
do a GROUP BY. I found one reference to a group_by() method, but
I don’t think the patch has been submitted or approved.

I spent some time trying to work out the right way to do it using the
model API, before realizing I was banging my head on the wrong brick
wall. I was using values() and [STRIKEOUT:select()]
extra(select=’…’) together, which should have clued me in right
away. Well, that and the fact that the data I was retrieving wasn’t one
of my model classes.

Using a tip from page 325 of the new django book (see my earlier
post
for a review), I decided to just go ahead and run the query
directly and work with the result set. The results need to go into a
template, so I wanted to have dictionaries instead of the tuples I get
from cursor.fetchall(). This is what I came up with:

from itertools import *
from django.db import connection

def query_to_dicts(query_string, *query_args):
    """Run a simple query and produce a generator
    that returns the results as a bunch of dictionaries
    with keys for the column values selected.
    """
    cursor = connection.cursor()
    cursor.execute(query_string, query_args)
    col_names = [desc[0] for desc in cursor.description]
    while True:
        row = cursor.fetchone()
        if row is None:
            break
        row_dict = dict(izip(col_names, row))
        yield row_dict
    return

You call it like:

results = query_to_dicts("""
    SELECT
        (year/10)*10 as decade,
        count(*) as case_count
    FROM
        docket_case
    GROUP BY
        decade
    """)

or, with arguments (this query counts the cases in each year of a
particular decade):

results = query_to_dicts("""
    SELECT
        year,
        count(*) as case_count
    FROM
        docket_case
    WHERE
        year/10 = %s
    GROUP BY
        year
    """, decade/10)

I don’t expect a large result set, but I figured this was an excuse to
experiment with generators and iterators. I’m pretty happy with the
results, but surprised I had to write something like this myself. I
didn’t see a cursor method to fetch the rows as dictionaries instead
of tuples, and the search I ran didn’t return anything that looked
useful.

Am I missing something?

[Updated 31 Dec: One correction: The method is extra(), the
argument is select. I was adding a count field using extra(), and
then asking for the value with values(). There does not seem to be a
way to structure the query I wanted using the models API methods.]

Django with PostgreSQL on Mac OS X Tiger

Most of the time, when I have worked on django projects, I’ve used the
SQLite backend for development on my PowerBook and deployed to
PostgreSQL on my Linux server. For the project I’m working on right now,
though, that turned into an issue when some of the queries that ran fine
on my dev system didn’t work at all on the production box. Apparently
the backend code responsible for assembling the SQL query strings was
producing different text for SQLite and PostgreSQL. So I could avoid
similar issues in the future, I set out to install PostgreSQL on my
laptop today.

Installing PostgreSQL itself turned out to be very easy indeed. I
downloaded the universal installer from Andy Satori’s “PostgreSQL for
Mac
” site. Some of the GUI clients included don’t work because I’m
on a PPC PowerBook instead of an x86 MacBook or MacBook Pro, but that’s
OK. I can use the CLI tools, which work fine.

The next thing I needed to do was set up psycopg. That turned out
to be a bit of an issue, since initd.org is having some sort of server
problem on their site. I was eventually able to download the tarball
with the sources for psycopg 1.1.21.

In order to compile psycopg, I also needed mxDateTime from eGenix.
They offer several pre-compiled packages, but none would install for me.
Working from the source for 3.0.0, I was able to compile it myself via
python setup.py install” into my virtualenv sandbox.

Back in the psycopg build directory, I was able to use these
instructions
, but had to hack around a bit to get the mxDateTime
headers in a place that matched the psycopg build expectations. I tried
several variations of path names into the mx source tree, but eventually
gave up and copied them all to one directory:

$ cd egenix-mx-base-3.0.0$ mkdir include$ find . -name '*.h' -exec cp {} `pwd`/include/ ;

I then tried to configure psycopg with:

$ ./configure --with-postgres-libraries=/Library/PostgreSQL8/lib --with-postgres-includes=/Library/PostgreSQL8/include --with-mxdatetime-includes=../egenix-mx-base-3.0.0/include/

That failed to find the Python.h header until I ran configure outside
of my virtualenv environment, using the copy of Python 2.5 I had
installed ages ago from python.org. Obviously your path to the mx
includes may vary, but that installer package for the PostgreSQL server
will put everything in /Library/PostgreSQL8.

Once I had configure running, I ran make (still outside of my
virtualenv). The build succeeded, and then I went over to the shell
running my virtual environment to install from there (via a simple “make
install”).

The end result of all of that was PostgreSQL 8.2.5 installed globally,
and the mx 3.0.0 and psycopg 1.1.21 packages installed only in my
virtual environment.

After a quick createdb, and edit to my settings.py file, I was
able to sync up my dev server against the new database and get back to
work. I suspect, but can’t verify, that I would have had fewer issues if
I was on an x86 Mac of some sort or running Leopard, since many of these
packages seem to have moved on ahead of my platform. The whole thing
took just over an hour, most of which was me fumbling around trying to
find compatible versions of the source for the various pieces since it
has been so long since I’ve compiled any of this stuff.

Book Review: The Definitive Guide to Django

I’m working on a new web-based project, and continuing the process of
learning django, so I was very pleased to receive my copy of The
Definitive Guide to django
by Adrian Holovaty and Jacob Kaplan-Moss in
the mail recently.

Contents

The book is divided into 3 sections. The first, including chapters
1-8, covers introductory material such as setting up a project, using
the template system and database layer, etc. This was familiar material
after several readings of the tutorials on the django web site, but
it has been cleaned up and organized nicely in the book.

The second section (chapters 9-20) cover “subframeworks” and dig
deeper into topics which, while documented online, I’ve found to be more
difficult to “discover”. The chapters on generic views, syndication and
site-map generation, and caching were particularly helpful. I also
appreciated the advice on production deployment in chapter 20. Some of
the mystery has been removed from these topics, and I learned about
features I didn’t even realize existed.

The last section, consisting of 8 separate appendixes, is a reference
manual for the various layers of django. All of the topics covered in
the main part of the book are included with more concise descriptions of
methods and more complete listings for functions or methods not
discussed earlier.

My Review

I’m glad I bought the book. It presents much of the same material you
can find online, but having it available in book form made it easier for
me to read without being distracted by trying the material at the same
time. I like to read, absorb, then try when I’m learning about a new
technology, and I find it much easier to read and absorb away from the
computer where there is no temptation to try writing code before I’m
really ready.

The writing is easy to read, but not dumbed down. Between the holidays
and the writing style of the book, I was able to blaze through the whole
thing in about a week’s time (I admit to skimming parts of the reference
section during that initial reading). There is a good mix of factual
information and best practices tips with arguments backing up the
opinions. You don’t have to agree with the suggestions, but you are more
informed after reading them.

I’ve been using the appendixes as a handy reference while working on
the templates and database queries for my project, and it has made
development quite a bit easier. The online references for django are
quite good, but flipping back and forth in the physical book is actually
quicker in a lot of cases.

Adam Gomaa explains why I prefer Django to Turbo Gears

I’ve built a couple of small web projects using Turbo Gears and
Django. I found the Django experience much nicer, in the end, and am
sticking with it for now. My main beef with Turbo Gears is that to
figure out how to do anything I had to visit 3-4 different web sites
to read documentation for the different pieces. The larger problem, as
Adam Gomaa points out in his post about Pylons and Django, is that
Turbo Gears lacked conceptual integrity.

“The ‘best tool for the job’ almost always consists of ‘the most
easily maintained tool for the job,’ which itself correlates
strongly with ‘what comes out the box.’ The fewer pluggable, add-on,
additional-setup-required components there are, the better. Two
templating languages is not a feature, it’s a mistake. Four? That’s
a nightmare.”

That nails it pretty succinctly. I don’t love working with Django.

They are slow to release new versions (even compatible versions) and
I’ve recently had to start tracking the trunk in order to build the
site I’m working on now (ugh). But at least all of the documentation
is in one place, and the various layers work together consistently.

two new releases

I’m working on another django project and need some head features for
it, so I needed to update my server from 0.96 to use an svn checkout (I
wish there were more frequent formal releases of django – they need a
release manager). As a result, I spent a while today updating a couple
of my applications to be compatible with the head version.

So, there are new (formally tested and released :-) versions of
django-links and codehosting available.