How I Review a PyCon Talk Proposal

Originally posted Oct 18, 2011 · 10 min read

As the submission period for PyCon 2012 comes to a close and we transition to reviewing the proposals from potential speakers, I thought I would talk about the criteria I use for evaluating the submissions. I am only one member of the Program Committee, and others may use different criteria, so please take that into account while reading this article.

I approach conference talk proposals in much the same way that I approached submissions for Python Magazine. Just as the magazine had a limited amount of space for articles each month, there are a limited number of speaking slots at PyCon. The Program Committee’s job is to choose the talks that maximize the benefit to the entire conference audience, within the space and time constraints given. While reviewing a proposal, I keep in mind that, as one person among 1,500, my opinion only goes so far toward the makeup of the conference audience. While my own enthusiasm and interest in a subject do matter, I try to consider the rest of the audience first. The main question I keep in my mind is, will the audience seeing this talk receive enough value to make it worth bumping another talk? A lot of factors go into that decision, and balancing them all can be a challenge.

Will the audience seeing this talk receive enough value to make it worth bumping another talk?

The Abstract

The first step to reviewing the proposal is to read it. That seems fairly obvious, but occasionally a title comes along that is so compelling that I have to remind myself to keep reading before voting +1 and moving on to the next talk. It isn’t enough for a proposal to cover an interesting topic. It has to indicate that the talk will be interesting, too. While I am reading, I look for several factors.

Is it clear, complete, and compelling?

First, is the abstract clear? The speaker should describe the topic they plan to talk about in terms I can understand, even if I don’t know anything about that subject area. The audience for PyCon is diverse in experience and background, and so are the speakers. A clearly written abstract, without a lot of domain-specific jargon, tells me the speaker will be able to communicate with the audience. If I can’t understand the proposal, I assume the audience won’t understand the talk either.

Next, is the abstract complete? An incomplete proposal is the largest negative factor I consider. If a proposal is incomplete, I can’t really say what the speaker will talk about, or even if they know the subject matter for their talk. If a proposal does not have a detailed summary or outline, I as the submitter to provide more detail.

Finally, I consider whether the abstract is compelling. Without regard to the actual subject, is the abstract written in a way to attract an audience? Is it full of boring clichés, vaguely worded assertions about the superiority of one tool over another, cute metaphors, or hand waving? I look for an abstract that shows the speaker is excited about the topic, and that they will be conveying that same excitement to the audience.

The Topic

For some people, the subject matter of a talk is the most important, or only, aspect taken into consideration when voting. I have seen presentations on topics I thought would be boring, but which were delivered with such enthusiasm that I enjoyed them more than talks I thought would be interesting from the outset. In my mind, the topic is an important factor, but not necessarily the most important, to be considered.

How relevant, niche, immediately useful, and novel is the topic?

I look first at whether the topic is relevant to the conference attendees. For PyCon, that largely means users of the Python programming language. Attendees will have a range of experience levels, interests, and backgrounds, though, so not every talk needs to include examples of Python code showing how to code around a problem. I have seen successful talks covering user interface standards compliance, community building, Java-based testing tools, and a host of other non-coding topics.

Although we want a broad set of topics, we do need to be careful to avoid talks that are too narrowly focused on a niche. PyCon has grown large enough that we run five tracks of talks simultaneously. That means each talk is usually competing with four others, and when the start and stop times don’t align it can be more than that. Each talk needs to attract enough of that audience to justify being included in the program instead of another of the many rejected talks. There are no formal metrics for that criteria, because we cannot know in advance how each talk will be attended. Talks that are unlikely to attract a significant audience at a large conference such as PyCon may find a better fit at a regional or subject-matter based conference. Examples of niche talks are those on topics that require deep technical knowledge of a scientific field, unpopular hardware platform, or industry. This is also why I tend to down-vote talks on brand new libraries or tools, usually with an admonition to resubmit the proposal when the project has matured and the community around it has grown. Now that PyCon includes a poster session, I often recommend that new projects which show a lot of promise convert their talk proposal into a poster proposal.

As a counterpoint to considering whether a topic is too niche, I also try to take into account whether the audience will take away something immediately useful. The members of the Python community blend open source and closed source solutions. Presentations on a closed source application or service are less interesting, unless the techniques presented can be applied elsewhere. Talks on open source projects are more likely to involve reusable code, just by their nature. However, new and incomplete projects don’t meet that standard.

The final criteria I consider for the proposal’s topic is whether it is novel. Although the PyCon has a significant amount of turnover in the audience year to year, I don’t want the program to be made up of the same topics over and over. The same goes for subjects that have appeared on blog posts, articles, and books. I am unlikely to up-vote an introductory talk for a project with excellent documentation unless the talk presents material in a new and interesting way.

The Presenter

Foremost in my mind when thinking about the presenter is the question of whether or not they will be successful at delivering their proposed talk. That question covers a lot of ground, so I break it up into several aspects and take each in turn.

How skilled, knowledgeable, and experienced is the speaker?

First, how skilled is the speaker? The PyCon audience deserves speakers who take their task seriously by preparing and practicing. Evaluating the presentation skills of an unknown speaker can be challenging. I have not had the opportunity to attend a lot of other conferences or user group meetings in person, so unless they have spoken at PyCon in the recent past I am unlikely to know much about their style or skill. Restricting myself to events I have attended means I may tend to favor a small group of “usual suspects.” To avoid that bias, I look for cases where they have spoken at user group meetings, regional conferences, or other events. Having access to videos of past presentations helps, and I try to review at least a few minutes of video for a speaker, when it is available.

I also want to see an indication that the speaker understands the subject well enough to speak at length about it. A good speaker does not need to be a domain expert, but some knowledge and experience is required. It is not necessary for every presentation on a tool to be delivered by the tool’s developer. Users and other community members with good public speaking skills can present

Finally, does the speaker have enough experience speaking to know their limits? There are two main limits: venue and time. PyCon is typically held in large hotel ballrooms and that type of venue presents a couple of challenges. The average audience for any given talk is 1/4 to 1/5 of the 1,500 conference attendees. Due to the structure of the venue, interactive sessions tend to flow less well than more the standard talk-followed-by-questions. Presentations that rely on showing code battle with the deep room layout, so fonts have to be large enough that big blocks of code don’t fit on a single slide. All of these issues can be overcome, but a presenter has to know that they can’t just slap up their text editor together with a terminal window using a low-contrast color scheme and expect the audience to follow along during a live demo. Previous speaking experience at smaller events shows me that a speaker is likely to understand these issues.

The time issue is more difficult to balance. I want to see a talk with enough depth that it shows something interesting, but reaching that depth may require a time-consuming introduction. Setting the audience level of the talk is one approach for avoiding this problem, because the audience for an Intermediate or Experienced talk will not expect as much of an introduction as someone in a Novice talk. On the other hand, assuming too much knowledge may mean even an Experience attendee does not understand the main point of a presentation. Speakers can reclaim time by removing redundant examples, tightening their focus, and trimming some of the commentary, so I usually try to point out material I think can be cut to give the speaker more time.

It is less common for an outline to seem too short to fill the space given, but it does happen. In this case, I look for information the presenter may need to add, suggest they work in extra examples, or simply tell them it looks too short. If a talk can’t fill a 30 minute slot, but is still important or interesting, I may propose they convert their proposal into a poster, instead.

Voting Versus Steering

One point frequently lost in the heat of the reviewing process is that the Program Committee is not just supposed to judge the proposals. Our responsibility is to make the PyCon program better. That mission includes guiding the speakers to improve their talks by refining the original proposals.

I have already mentioned a some of the common suggestions I make during the course of a review. After considering all of the other factors, I pause and pretend that the proposal I am reviewing is for the only talk I will be able to see at the conference. Then I think of ways the presentation could be improved, and engage with the speaker to discuss them. I try to provide guidance based on past experience to help newer speakers understand how their talk might be improved to fit into PyCon better.

I also tend to ask a lot of questions, especially if I or other reviewers have a negative impression of the proposal. Asking for clarification or more detail is an important part of the process. Sometimes the interaction with the speaker helps me decide whether to champion a talk, or merely vote +0 or -0.

tl;dr

As I said at the outset, other reviewers may use different criteria when evaluating a proposal. We want a big enough Program Committee that we have a range of experience and input into the selection process. But these are the criteria I use, so if I end up reviewing your proposal, you know how I went about it.

Is the abstract clear?
Is the abstract complete?
Is the abstract compelling?
Is the topic relevant?
Is the topic too niche?
Is the material immediately useful?
Is the presentation novel?
Is the speaker skilled?
Is the speaker knowledgeable?
Is the speaker experience?
How can the proposal/talk be improved?