As the submission period for PyCon 2012 comes to a close and we
transition to reviewing the proposals from potential speakers, I
thought I would talk about the criteria I use for evaluating the
submissions. I am only one member of the Program Committee, and others
may use different criteria, so please take that into account while
reading this article.
I approach conference talk proposals in much the same way that I
approached submissions for Python Magazine. Just as the magazine had a
limited amount of space for articles each month, there are a limited
number of speaking slots at PyCon. The Program Committee’s job is to
choose the talks that maximize the benefit to the entire conference
audience, within the space and time constraints given. While reviewing
a proposal, I keep in mind that, as one person among 1,500, my opinion
only goes so far toward the makeup of the conference audience. While
my own enthusiasm and interest in a subject do matter, I try to
consider the rest of the audience first. The main question I keep in
my mind is, will the audience seeing this talk receive enough value to
make it worth bumping another talk? A lot of factors go into that
decision, and balancing them all can be a challenge.
Will the audience seeing this talk receive enough value to make it
worth bumping another talk?
The first step to reviewing the proposal is to read it. That seems
fairly obvious, but occasionally a title comes along that is so
compelling that I have to remind myself to keep reading before voting
+1 and moving on to the next talk. It isn’t enough for a proposal to
cover an interesting topic. It has to indicate that the talk will be
interesting, too. While I am reading, I look for several factors.
Is it clear, complete, and compelling?
First, is the abstract clear? The speaker should describe the topic
they plan to talk about in terms I can understand, even if I don’t
know anything about that subject area. The audience for PyCon is
diverse in experience and background, and so are the speakers. A
clearly written abstract, without a lot of domain-specific jargon,
tells me the speaker will be able to communicate with the audience.
If I can’t understand the proposal, I assume the audience won’t
understand the talk either.
Next, is the abstract complete? An incomplete proposal is the
largest negative factor I consider. If a proposal is incomplete, I
can’t really say what the speaker will talk about, or even if they
know the subject matter for their talk. If a proposal does not have a
detailed summary or outline, I as the submitter to provide more
Finally, I consider whether the abstract is compelling. Without
regard to the actual subject, is the abstract written in a way to
attract an audience? Is it full of boring clichés, vaguely worded
assertions about the superiority of one tool over another, cute
metaphors, or hand waving? I look for an abstract that shows the
speaker is excited about the topic, and that they will be conveying
that same excitement to the audience.
For some people, the subject matter of a talk is the most important,
or only, aspect taken into consideration when voting. I have seen
presentations on topics I thought would be boring, but which were
delivered with such enthusiasm that I enjoyed them more than talks I
thought would be interesting from the outset. In my mind, the topic is
an important factor, but not necessarily the most important, to be
How relevant, niche, immediately useful, and novel is the topic?
I look first at whether the topic is relevant to the conference
attendees. For PyCon, that largely means users of the Python
programming language. Attendees will have a range of experience
levels, interests, and backgrounds, though, so not every talk needs to
include examples of Python code showing how to code around a
problem. I have seen successful talks covering user interface
standards compliance, community building, Java-based testing tools,
and a host of other non-coding topics.
Although we want a broad set of topics, we do need to be careful to
avoid talks that are too narrowly focused on a niche. PyCon has
grown large enough that we run five tracks of talks
simultaneously. That means each talk is usually competing with four
others, and when the start and stop times don’t align it can be more
than that. Each talk needs to attract enough of that audience to
justify being included in the program instead of another of the many
rejected talks. There are no formal metrics for that criteria, because
we cannot know in advance how each talk will be attended. Talks that
are unlikely to attract a significant audience at a large conference
such as PyCon may find a better fit at a regional or subject-matter
based conference. Examples of niche talks are those on topics that
require deep technical knowledge of a scientific field, unpopular
hardware platform, or industry. This is also why I tend to down-vote
talks on brand new libraries or tools, usually with an admonition to
resubmit the proposal when the project has matured and the community
around it has grown. Now that PyCon includes a poster session, I often
recommend that new projects which show a lot of promise convert their
talk proposal into a poster proposal.
As a counterpoint to considering whether a topic is too niche, I also
try to take into account whether the audience will take away something
immediately useful. The members of the Python community blend open
source and closed source solutions. Presentations on a closed source
application or service are less interesting, unless the techniques
presented can be applied elsewhere. Talks on open source projects are
more likely to involve reusable code, just by their nature. However,
new and incomplete projects don’t meet that standard.
The final criteria I consider for the proposal’s topic is whether it
is novel. Although the PyCon has a significant amount of turnover in
the audience year to year, I don’t want the program to be made up of
the same topics over and over. The same goes for subjects that have
appeared on blog posts, articles, and books. I am unlikely to up-vote
an introductory talk for a project with excellent documentation unless
the talk presents material in a new and interesting way.
Foremost in my mind when thinking about the presenter is the question
of whether or not they will be successful at delivering their proposed
talk. That question covers a lot of ground, so I break it up into
several aspects and take each in turn.
How skilled, knowledgeable, and experienced is the speaker?
First, how skilled is the speaker? The PyCon audience deserves
speakers who take their task seriously by preparing and practicing.
Evaluating the presentation skills of an unknown speaker can be
challenging. I have not had the opportunity to attend a lot of other
conferences or user group meetings in person, so unless they have
spoken at PyCon in the recent past I am unlikely to know much about
their style or skill. Restricting myself to events I have attended
means I may tend to favor a small group of “usual suspects.” To avoid
that bias, I look for cases where they have spoken at user group
meetings, regional conferences, or other events. Having access to
videos of past presentations helps, and I try to review at least a few
minutes of video for a speaker, when it is available.
I also want to see an indication that the speaker understands the
subject well enough to speak at length about it. A good speaker does
not need to be a domain expert, but some knowledge and experience is
required. It is not necessary for every presentation on a tool to be
delivered by the tool’s developer. Users and other community members
with good public speaking skills can present
Finally, does the speaker have enough experience speaking to know
their limits? There are two main limits: venue and time. PyCon is
typically held in large hotel ballrooms and that type of venue
presents a couple of challenges. The average audience for any given
talk is 1/4 to 1/5 of the 1,500 conference attendees. Due to the
structure of the venue, interactive sessions tend to flow less well
than more the standard talk-followed-by-questions. Presentations that
rely on showing code battle with the deep room layout, so fonts have
to be large enough that big blocks of code don’t fit on a single
slide. All of these issues can be overcome, but a presenter has to
know that they can’t just slap up their text editor together with a
terminal window using a low-contrast color scheme and expect the
audience to follow along during a live demo. Previous speaking
experience at smaller events shows me that a speaker is likely to
understand these issues.
The time issue is more difficult to balance. I want to see a talk with
enough depth that it shows something interesting, but reaching that
depth may require a time-consuming introduction. Setting the audience
level of the talk is one approach for avoiding this problem, because
the audience for an Intermediate or Experienced talk will not expect
as much of an introduction as someone in a Novice talk. On the other
hand, assuming too much knowledge may mean even an Experience attendee
does not understand the main point of a presentation. Speakers can
reclaim time by removing redundant examples, tightening their focus,
and trimming some of the commentary, so I usually try to point out
material I think can be cut to give the speaker more time.
It is less common for an outline to seem too short to fill the space
given, but it does happen. In this case, I look for information the
presenter may need to add, suggest they work in extra examples, or
simply tell them it looks too short. If a talk can’t fill a 30 minute
slot, but is still important or interesting, I may propose they
convert their proposal into a poster, instead.
Voting Versus Steering
One point frequently lost in the heat of the reviewing process is that
the Program Committee is not just supposed to judge the
proposals. Our responsibility is to make the PyCon program
better. That mission includes guiding the speakers to improve their
talks by refining the original proposals.
I have already mentioned a some of the common suggestions I make
during the course of a review. After considering all of the other
factors, I pause and pretend that the proposal I am reviewing is for
the only talk I will be able to see at the conference. Then I think of
ways the presentation could be improved, and engage with the speaker
to discuss them. I try to provide guidance based on past experience to
help newer speakers understand how their talk might be improved to fit
into PyCon better.
I also tend to ask a lot of questions, especially if I or other
reviewers have a negative impression of the proposal. Asking for
clarification or more detail is an important part of the process.
Sometimes the interaction with the speaker helps me decide whether to
champion a talk, or merely vote +0 or -0.
As I said at the outset, other reviewers may use different criteria
when evaluating a proposal. We want a big enough Program Committee
that we have a range of experience and input into the selection
process. But these are the criteria I use, so if I end up reviewing
your proposal, you know how I went about it.
- Is the abstract clear?
- Is the abstract complete?
- Is the abstract compelling?
- Is the topic relevant?
- Is the topic too niche?
- Is the material immediately useful?
- Is the presentation novel?
- Is the speaker skilled?
- Is the speaker knowledgeable?
- Is the speaker experience?
- How can the proposal/talk be improved?