PyWorks 2008 Nov 12-14

MTA is putting the finishing touches on our plans to host the first
annual combined PyWorks and php|works conference Nov 12-14. This
year’s conference builds on the past success of php|works events by
adding 2 new tracks of presentations for Python developers and a
cross-over track with topics of interest to everyone.

The tutorials on Wednesday offer an excellent introduction to four of
the popular frameworks available for Python web applications. Mark Ramm
promises a fast paced introduction to TurboGears and its related
components. Travis Cline will be presenting Django from the perspective
of a PHP developer, comparing and contrasting the approaches used by
both. Brandon Rhodes’ Grok tutorial is sure to create a few Zope
converts, and Noah Gift will show how to build and deploy AJAX
applications on Google’s AppEngine platform.

The keynote Thursday morning will be given by Kevin Dangoor, well
known in the Python community for his leadership of the TurboGears
project. Kevin will talk about the aspects of running successful open
source projects aside from the code itself. Managing developers is
challenging enough, but when your project is run by volunteers there are
unique aspects to consider. He will offer suggestions for how to
increase and manage the participation in a project based on his
experiences with the TurboGears project.

Illustrating the broad spectrum of applications areas where it is
used, the subjects of the Python talks on Thursday and Friday include
topics such as language internals, artificial intelligence, user
interface tools, scaling multi-threaded applications, and systems
administration. Of course, there’s a healthy dose of web development as
well.

I’m really looking forward to the the conference, and I hope to see
you there!

Python Magazine for October 2008

The October 2008 issue of Python Magazine is available for download
now.

Jan Dittberner’s cover story on sqlalchemy-migrate explains how to
manage changes to your database schema.

Learn about building applications on Google’s AppEngine from Kumar
McMillan.

Doug Farrell shares a Twisted idea for logging from distributed
applications.

And Drew Smathers explains some of the intricacies of multiple
inheritance in Python, including the MRO and super().

Jesse Noller’s column shows us a completely different way to use SSH
directly from Python via Paramiko.

In his Welcome to Python column this month, Mark Mruss explains how to
use variable arguments to functions.

And to wrap up the issue, Steve looks at some of the important events
that have occurred in the Python world recently, including major
releases of Django 1.0 and Python 3.0 as well as the PSF Community
Awards.

Grab your copy today!

To Build or To Buy?

Recently a friend sent me a link to an old post on Rick Copeland’s
blog titled Three Reasons Why You Shouldn’t Write Your Own Web
Framework
. Although
Rick talks specifically about open source web frameworks, his
comments raised the more general question of how should we draw
the line when deciding whether to build a new anything versus
“buying” or using an existing project. The answer is not always
clear, because it depends on a lot of factors.

Rick’s arguments against starting from scratch are primarily about
avoiding wasting time, yours and other developers’. You end up
wasting your time because you’re always playing catch-up, implementing
features and fixing bugs that have already been handled by other
tools, instead of building the features your customers wanted in the
first place. You waste other developers’ time because you add to
their burden of evaluating the solutions available in a larger pool of
options. Although the cliché example for Python developers is the
over-abundance of web frameworks, the situation with template
languages
is even worse.

So why do people start new template languages, web frameworks, ORMs,
GUI toolkits, et cetera, instead of using and contributing to an
existing project? Before open source became so prevalent,
contributing to an existing project wasn’t really an option, so the
economic considerations for building vs. buying were straightforward:
Could you build the thing, either via contractors or full-time
employees, for less than it would cost to buy it from a vendor, adapt
it to your needs, and pay for the support? These days, open source
eliminates a lot of the questions about up-front cost, but that
doesn’t necessarily make the question any easier to answer.

For most open source projects, there isn’t anyone to pay for support
and, as Titus Brown points out in his post It’s just too easy, “It’s
less work to code a few hundred lines of Python than it is to
understand someone else’s few hundred lines of Python”. That’s
where we fool ourselves, though. Developers tend to have tunnel
vision, seeing the easy parts of a project clearly and glossing over
the tricky edge cases that are going to trip us up. Starting from
scratch may only look like less work because we don’t see the
hard parts.

On the other hand, sometimes starting from scratch can have benefits.
For one thing, picking up in the middle of a project can be difficult,
too. There’s a longstanding joke in the Zope community that its
learning curve is “Z-shaped”. A few years ago I made the effort to
learn its inner workings and was a serious Zope user. But when I put
it down and stopped using it regularly, my skills became rusty very
quickly. Like the saying goes, “it’s a great place to live but I
wouldn’t want to visit there”. Maybe that says more about me than
Zope, though. In any event, a smaller framework may do less, but can
be easier to pick up and learn. Fewer features also mean less stuff
to keep in your head once you become an experienced user.

“it’s a great place to live but I wouldn’t want to visit there”

An obvious reason to build your own is if none of the existing tools
do what you want. When I built HappyDoc, back in 1998, I only started
the project because the other similar doc-extraction tools had a
limitation that made it impossible for me to use them: They all
imported your code to find the classes, functions, and docstrings.
Unfortunately, the code I wanted to document had import side-effects,
so generating the documentation with tools that imported it required a
special server configuration with resources that weren’t easy to
replicate. The key feature of HappyDoc was that it worked by reading
and parsing the source code, without doing a full “import”, so there
were no side-effects. Since then, other tools have come along that
work the same way, but at the time it was enough to differentiate
HappyDoc from the other contenders. I ended up writing more than a
few hundred lines of Python, though, so, re-examining my decision with
the benefit of hindsight, I see that I might have been better off
contributing to an existing project instead of starting a new one.

There’s still no easy way to answer the build-versus-buy question. I
lean toward using open source projects when possible, but there are
times when you’re just better off building exactly what you want than
adapting your code to someone else’s way of thinking. How do you
decide when to build and when to buy? Let me know what you think.

Originally published in Python Magazine Volume 2 Issue 10 , October, 2008

Python Magazine for September 2008

The September 2008 issue of Python Magazine is available for
download now.

The cover story, django at 30,000ft, comes from Justin Lilly and
coincides (as closely as we could make it) with the release of Django
1.0. Justin provides a high-level overview of the features of Django for
anyone evaluating it, or convincing their manager to do so.

Also up this month is Duplicate Code Detection Using Clone Digger,
in which Peter Bulychev talks about tools and techniques for finding
repetitive sequences in your code, even if they don’t match exactly.
Clone Digger looks like a great utility to add to your toolbox to
reduce your code base and keep it DRY.

JC Cruz returns with another PyObjC and Cocoa article, Displaying
Tabular Data with PyObjC
. JC covers everything you need to know to
start using the NSTableView class in your own programs.

We also have 2 community reports this month. First, Alex Holkner walks
us through his experience with PyWeek and building a brand new game in
one week using pyglet. Then Rodrigo Bernardo Pimentel talks about the
São Paulo Python user group’s experience hosting their own hack-a-thon
for Python Bug Days.

In Welcome To Python this month, Mark Mruss covers the slicing
API for manipulating sequences. As usual, Mark’s explanation is clear
and easy to follow.

For our technical column, And Now For Something Completely
Different
, Rick Harding introduces the Bazaar distributed version
control system and talks about a few of its key features.

Steve Holden talks about OSCON, the status of Python 2.6 and 3.0,
and Python certification.

In my own column, Django’s move to 1.0 leads me to contemplate the
meaning of “release” and whether it is better to do it “early and often”
or “regularly”.

Grab your copy today, and as always direct questions or comments to
“doug dot hellmann at pythonmagazine dot com” or in comments on this
blog.

Releasing Software

“Release early, release often.” How often is enough and does it
count if you just commit?

I’ve always been interested in development processes and how they
impact the software being created. I have worked in shops with a
range of formal processes, and after some experimentation with
different methodologies, I find myself most comfortable somewhere in
the middle of the spectrum. I like to have enough controls in place
to understand how forward progress is made and prevent regressions,
but also want a certain degree of flexibility to bend the rules when
the situation calls for it. While no software is ever complete, I
want the ability to put a stake in the ground and say “This version
is done”, for some definition of “done”.

In early June, Christian Joergensen wrote on his blog about his
frustrations with the lack of official releases from Django
.
Until recently, the Django team has had a policy of releasing by
committing to the trunk in their subversion repository, but going no
further until they were ready for the 1.0 release. Producing
intermediate releases was deemed to take time and effort that could
otherwise be applied to development of new features, refactoring, API
polishing, and all of the other work that was on the “must have” list
for 1.0. As a result, the active Django user community has grown used
to “tracking trunk”, to the point that the This Week in Django podcast has an entire segment
dedicated to reporting trunk changes.

In his post, Joergensen made several points about the difficulties of
keeping up with security patches and managing configurations on
multiple servers for different projects. The ensuing discussion in
the comments, and later on the django-dev mailing list, were
interesting; he seems to have struck a nerve. The reactions ranged
from heated rejections of the idea to whole-hearted agreement that a
release was long over due. The entire conversation was a little
surprising to me, because I’ve never really considered not releasing
software as part of a development cycle, even for my personal open
source projects. SVN repository access has always been a bonus on top
of formal releases.

Because I’m a Django user but find myself without the time (or
inclination) to “track trunk”, I was glad to see that the Django team
used the discussion resulting from Joergensen’s post as the impetus to
ramp up their efforts to wrap up the 1.0 release. As I write, the
major branches have been merged into trunk and an alpha release is
available, with a beta due to be posted soon. The final 1.0 release
candidate should be online by the time you read this. Reaching 1.0 is
a big step for any project, and especially one of this size.
Congratulations, Django developers!

The conversation led me to think about the fact that in the commercial
software world producing releases is an unquestioned activity. What
makes commercial and free apps different in this respect? Is it that
users don’t have access to the source code? That’s not always the
case, depending on the way the commercial application was built. Some
“enterprise” applications are big (and complicated) enough that having
access to the source is not, by itself, sufficient to let you
replicate their functionality. In these cases, at least part of the
source is distributed so that services teams can modify it when
integrating it with other systems.

Some might point out that companies wouldn’t be paid if there was no
release. That’s not always true, either. Over time, companies tend
to make more from support contracts than from selling the initial
licenses to the application. Professional services for installation
and customization can also be big source of revenue. In quite a few
cases, the services revenue is enough to support the development of
open source software that is freely licensed. See Zope Corp., Sugar
CRM, and Collabnet for a few examples of companies that have organized
creative business models around open source software.

Perhaps it is a fundamental difference in the definition of what is
being created. Many organizations don’t even consider software a
product until there is documentation and training materials to go
with it. Preparing an official release is an important convergence
point for the different parts of the project team. The QA,
documentation, and professional services groups all have deliverables
of their own to be added to the software. The act of preparing the
release forces everyone involved to tie up loose ends from incomplete
features (or at least tuck them under).

Overall, I come down on the side of formal releases. The steps of
packaging a set of files to create a release tells the end user that
the results are “official”, in some sense. New and casual users of a
project, who don’t have time to follow every commit log message or
keep up with the developer mailing list to understand when it is safe
to pull a version out of the source repository, need that affirmation
when they are trying to figure out how to install the application.
I’ve also encountered plenty of production shops where software could
only be installed through the OS native package management system,
making packaged releases a requirement.

There are plenty of questions left open about how frequently to
prepare releases, and how to define them. We have to balance features
with schedule. Lacking economic incentive to create regular releases
to keep maintenance contract fees coming in, open source packages tend
to place more emphasis on features, letting the date slip as needed.
That’s as it should be. On the other hand, corporate customers like
regularly scheduled releases because they make planning easier. With
hundreds, or thousands, of systems to keep consistent, they can’t
update them piecemeal or have patches coming faster than they can test
and apply them. Even if the software runs on only a few
infrastructure systems, a certain level of stability in the day-to-day
operation of the company is necessary. Life is chaotic enough without
having to worry whether the email server is going to be online every
day.

Instead of “release early, release often”, should we be saying
“release early, release regularly”? Let me know what you think in
the comments.

Originally published in Python Magazine Volume 2 Issue 9 , September, 2008

Python Magazine for August 2008

The August 2008 issue of Python Magazine is available for download
now.

The cover story this month is from Greg Pinero and talks about
http://www.utilitymill.com/. If you don’t know about Utility Mill, you
should check it out. I don’t think there’s an easier way to create a web
utility. Greg explains some of the security and performance issues he
encountered creating the site and walks you though posting your own app.

In Drawing Presentable Trees, Bill Mill describes the evolution of
tree rendering approaches, with plenty of graph traversal algorithms for
us comp sci geeks.

Juri Pakaste explains how to make your code more loosely coupled in
Using Dependency Injection in Python. I’m looking forward to applying
these ideas in some code I’m working on right now to make it easier to
test.

And wrapping up the feature list this month is Paul McGuire with
Advanced Pyparsing: Implementing a JSON Parser Using Results Names.
Paul shows how to parse complex structures and extract only the useful
information without twisting yourself into knots.

Jesse Noller’s column recounts some of his experiences getting into
Python core development. He gives a brief tour of the code base and
offers tips to avoid some of the issues he ran into.

Mark Mruss looks at the wxWidgets and wxPython libraries for creating
GUI application. And Steve Holden introduces the Board of Directors of
the Python Software Foundation and talks about the role of the PSF.

Grab your copy today, and as always direct questions or comments to
“doug dot hellmann at pythonmagazine dot com”.

PyWorks 2008 Conference Schedule

Brian has posted the schedule for PyWorks 2008. He has put
together a good mix of topics, including the cross-over with the
php|works side of the conference. The presenter list includes some
familiar names, and some new ones, too. Now if I could only figure out
how to be in 2 places at the same time, so I could listen to Kevin
Dangoor talk about ZODB and Jacob Taylor’s presentation on artificial
intelligence. Maybe Guido will let me borrow the time machine if I ask
nicely.

Join us in Atlanta on Nov 12-14, 2008 for the first annual `PyWorks`_
conference.

Community Participation

Originally published in Python Magazine Volume 2 Issue 8 , August, 2008

Our authors are having an impact through their participation in
the global Python community, and we couldn’t be more proud to
support them.

One of my main motivations for becoming involved with Python Magazine
was to contribute back to the Python community that has provided so
much help to me in the past. I’ve had access to innumerable open
source programs and libraries, learned about algorithms and
development techniques from some of the best programmers around, and
received a good share of support in the form of bug reports and
patches for my own projects. All of that came from the Python
community, found first on Usenet, then mailing lists, and now blogs.
In return for all I’ve received, I want to turn this magazine into a
tool for exploring new ideas, promoting user group participation, and
supporting Python-based projects with exposure and publicity. I’m very
happy to say that, so far, those goals seem to be coming together
quite well.

MTA and Python Magazine are committed to supporting user groups and
encouraging active participation. I mentioned last month that we have
been running a subscription promotion, tied to user group membership.
The response has been excellent, with groups from all over the world
signing up. We’re also providing a user group discount rate to PyWorks
in November. There is an opportunity to do more to help encourage
growth of these groups, though. The community needs to develop ways to
exchange ideas for activities, presentations, and projects, and we
will be working with group organizers to foster that sort of
communication over the coming months. There is already a public
accessible list of local user groups on the python.org wiki, and a
blog for publicizing group activities, but I’m interested in hearing
from you about what sorts of support you think would be helpful. Are
the biggest needs tools? Organizers? Or something else entirely?

Writing about Python and starting these sorts of discussions is not
the only way contributors to the magazine participate in the
community, of course.

Another of our missions is to publicize projects that are useful,
especially when more participation may be needed. The June cover story
by Grig Gheorghiu on Pybots is an excellent example. That article
caused Glyph Lefkowitz to initiate a discussion on the python-dev
mailing list about cleaning up the existing bots, and how to best
handle feedback from failed builds. The consensus was to have project
and individual bot maintainers report issues as they come up,
distributing the work of monitoring the tests to those most interested
in individual test cases. Check out the python-dev thread Community
buildbots and Python release quality metrics
for more details,
especially if you are able to help by running or monitoring a bot of
your own. I spent part of the last week setting up buildbot (the basis
of the pybots project) at my day job to run our own internal tests. To
my surprise, it has been a lot easier to configure than I thought it
would be given the number of features it provides, and I already have
a swiftly growing list of other tasks we want to automate with it.

Writing about Python and starting these sorts of discussions isn’t
the only way contributors to the magazine participate in the
community, of course. Most of us are programmers by day and contribute
to open source projects in our free time. This month, Associate Editor
Jesse Noller’s column recounts what he has learned about Python core
development from his efforts to integrate the multiprocessing
module (formerly known as processing) with the standard
library. Long time subscribers will have noticed a recurring theme in
our coverage of using threads vs. processes for parallelizing
applications. We have discussed the topic in several forms over the
past year, including explanations of the GIL and its impact as well as
several techniques to overcome perceived limitations. Because of his
background in massively parallel load testing Jesse has a lot
experience in this area and he was an excellent advocate for PEP 371,
“Addition of the multiprocessing package to the standard library”.
His hard work has paid off now that the new module is included in the
latest release of Python 2.6. Congratulations to Jesse and
processing‘s original author Richard Oudkerk!

If you have suggestions for ways Python Magazine can help your local
users group, or you just want to comment on or ask questions about an
article or column you’ve seen in Python Magazine, post a note in the
comments below.

Python Magazine for July 2008

The July issue is available for download now.

Brandon Rhodes delivers another cover story for us, this time on
Kinetic Style Sheets. KSS is a JavaScript engine that interprets files
full of CSS-like instructions that can launch both client-side and
server-side actions in response to the user’s typing and clicks. Brandon
makes it look easy, as usual.

Speaking of easy, Jonathan LaCour takes all of the mystery out of
metaclasses with an explanation clear enough to make me reconsider using
them in my own work.

This month also brings us the final installment of Learning Python
with PyGame
from Terry Hancock. The Cat and Mouse game comes together
with sprites and more animation to create something ready to play.

Once you’ve finished learning PyGame, let Richard Jones inspire you to
participate in PyWeek, a semi-annual game programming challenge using
Python. Richard’s journal walks through the evolution of his own game,
and shows how much fun the contest can be.

Threaded Comments for Your Site from Eric Florenzano rounds out
the feature articles for July. If you’re building a web site using
Django and want to support discussions or comments, check out Eric’s
article and django-threadedcomments.

In his Welcome to Python column, regular contributor Mark Mruss
introduces the set data type. Jesse Noller “gets with” context
managers in Completely Different, exploring the new feature available
in Python 2.5’s __future__ module. Finally, Steve Holden considers
whether scientists and technologists should be more involved in social
and political issues.

Enjoy!