Planting Acorns
This post is based on the closing keynote I gave for PyTennessee in February 2018, where I talked about how the governance of an open source project impacts the health of the project, and some lessons we learned in building the OpenStack community that can be applied to other projects. The presentation was not recorded.
OpenStack is a cloud computing system written in Python. The project is 8 years old. During that time we have had 18 major releases, and to give you an idea of the size of the community, during 2017, we had 2,500 people contribute more than 65,000 patches. The age and size of the project means our community has been through several transitions and challenges that other smaller projects may not yet have encountered, and my hope is that by thinking about them early, you can prepare for them, and your communities will be able to grow in a healthier way.
I started working on OpenStack in 2012, 2 years after the project had been launched and right at the beginning of a period of increased interest and contributions. It was the leading edge of the up-slope for our hype curve. I started looking at the code with my team at Dreamhost, but my first significant exposure to the community was at a Design Summit, an in-person event for organizing the teams working on different parts of the software. It was an energizing experience.
I had been to conferences before, including open source conferences like PyCon and PyTennessee, but never anything quite like this. We had hundreds of contributors gathered together in one place for a week of meetings, discussions, arguments, decision making, and, just as importantly, socializing. The enthusiasm of the participants was amazing, and the event led me to dive deeper and deeper into the project over the following 6 years.
At that summit I was part of starting two new teams, one based on metering cloud use for billing and one based on building a set of reusable libraries for the different OpenStack services. My experience with trying to help launch those teams within the community sparked my ongoing interest in open source project governance — not just how to build open source software, but how to build the communities around it — in part because, at the time, OpenStack had no system in place for expanding the scope of the project, and the community struggled to work out how to do it. Today, I want to share 9 themes that may help you avoid pitfalls when considering governance in your own projects.
1. Think Long Term
About 15 years ago, I learned about a group called the Long Now Foundation, which was established in 1996 to foster long-term thinking. They run an interesting series of talks along with several other projects. The most widely publicized of those is a clock that will tell time for 10,000 years.
Stop and think about what that means. 10,000 years.
I love the audacity of that project. That is seriously long-term thinking.
In describing the project, Danny Hillis, the primary designer of the clock said:
I cannot imagine the future, but I care about it. I know I am a part of a story that starts long before I can remember and continues long beyond when anyone will remember me. I sense that I am alive at a time of important change, and I feel a responsibility to make sure that the change comes out well. I plant my acorns knowing that I will never live to harvest the oaks.
— Danny Hillis
I hope no one is running any of my code in a century, much less ten millennia, but the spirit of what Hillis said has stuck with me, and I try to apply it to my open source work.
The whole concept of open source is still in its early days. Sharing source code publicly started with BSD in the ’70s; Stallman’s work on the GPL was in the ’80s; and we’ve only recently celebrated the 20th anniversary of the term “open source,” coined in the late ’90s. So it is spreading and growing, but it’s still quite a young idea.
The product of that idea has two parts, the software and the community around it. The software we are building today is ephemeral; that’s the nature of technology. But the community we are building, and the precedents we are setting along the way, could outlive us if we do our work with care. To ensure that it does, we need to be deliberate about how we structure our communities, and especially how we set up governance.
Today, more than 8 years into the project, we are still arguing about what OpenStack should be, due to the narrow definition of “Infrastructure as a Service” adopted early on. What exactly does “infrastructure” mean? Where in the stack does it stop? That ambiguity introduced stress and challenges when we tried to expand our scope around the time I joined. The organization had been set up with short-term thinking about what OpenStack itself was and should be, based on the environment in which it was created, rather than what it might become.
OpenStack started when teams building two tools joined together, providing an object storage system (Swift) and a compute orchestration system (Nova).
Over time, features of the compute orchestration system were pulled out into their own services, and teams formed around them to focus on the features those services provided, like identity management, image management, block storage, etc.
The metering project that I was part of was brand new, rather than coming out of the compute project. That meant the community had a lot of conversations about whether it should really even be part of OpenStack, or if it should be part of a larger ecosystem of projects, to be managed outside of the set of official projects.
We initially solved the community growth and structure problem by designating feature-based program areas such as compute and telemetry, to be owned and managed by different teams. However, this quickly led to conflict when two groups of people wanted to solve the same problems in different ways.
For example, we spent a lot of time trying to integrate three or four different tools related to telemetry, for metering and monitoring, with different implementations. And we completely failed. The approaches of the teams were just too different, and there was no good way to integrate the technology or, more importantly, the goals of the teams.
2. Organize Around People
When we considered the fact governance is not about technology, it’s about people, we realized we would need to change our community’s structure if we were going to support work on projects that competed with each other.
Other communities have similar competition; just look at how many web frameworks we have within the Python community. But initially, it felt awkward to many of us to have competing projects be part of what we were trying to view as a single product. However, we came to see that in order to include all of those new teams of people in our community, and encourage them to work in the open without hindering their creativity, we needed to focus on the people rather than feature areas.
So we did away with program areas and rewrote our rules about feature overlap so that competing projects could be supported as long as they had a clear mission that supported the overall OpenStack mission.
This change let us add quite a few new projects, not just in the metering and monitoring areas. Teams building different deployment tools joined the community, and their joint participation has helped us understand ways to simplify and improve deployment and meet the needs to an ever-growing user base. All of the new teams were able to understand what they were committing to do and what the community was committing to support in return.
Many other factors contributed to people’s interest in joining OpenStack, but this focus on community is still an important aspect that has led to our continued growth. As a result, the OpenStack Foundation has begun to explore ways to expand our scope even further by accepting projects not related to the original infrastructure-layer mission in areas like machine learning, edge computing, and networking — all areas that intersect with the original mission, but are not clearly part of the traditional definition of “infrastructure.”
3. Document Your Culture
As we expanded our scope and attracted more and more people, we experienced growing pains. The change of approach that allowed us to move past the challenges of competing projects is just one of many examples of areas where we needed to reconcile different perspectives from different sides of our community. As we have tackled each one, we have come back repeatedly to a central issue of needing a shared culture.
It’s easy to start a new project by throwing some code up on Github, but as it starts attracting new contributors, they need to share an understanding of what makes your community hold together. Word of mouth works for small groups, but as your community grows, it becomes more and more important to write these things down. It’s a big planet and not everyone is available for synchronous interactions, whether in person or online. Documenting community expectations lets your community grow beyond the bounds of a small group who all know each other.
Those cultural documents need to go beyond the mechanical aspects of having contribution guidelines and the license you use. Many projects have a Code of Conduct to address unwanted behavior, but I’m talking about answering bigger, more fundamental, questions like What is the mission of this community? and What brings us together?
The OpenStack community has two documents describing our core principles. We started with a list of what we call The Four Opens:
- Open Source — We use the Apache 2.0 license and there are no extra features available elsewhere. We are not “open core”. There is no “enterprise edition”.
- Open Design — Planning and feature selection are driven by the community, together. There is no secret cabal or driving company.
- Open Development — The code is published via public git repositories. Patches are reviewed publicly. Planning documents are published in accessible formats.
- Open Community — Our leaders are elected. We follow what we call a lazy consensus model for decision making, settling questions based on lack of strong objections rather than requiring strong buy-in. All of our community processes are documented.
This set of principles served us well for some time. We were able to discourage anti-community behavior like companies developing projects privately and then dumping project source on the community, or trying to take complete control of a project.
But as we grew and had new leaders emerge, we realized that not everyone understood those short, pithy four opens in the same ways. That led to misunderstandings about some decisions. So, we produced another more detailed list of our Guiding Principles. I will talk about a few of those today, but the important insight was to write them down, and to use the list to build and reinforce a shared culture.
4. Build Consensus
One of the most important principles we have is to use consensus for decisions. Relying on consensus can be frustrating for people who want to move fast, but it is important to the long term success of a project. Managing the interests of all of our contributing companies is a good example of this.
Many of our contributors are paid to work on the project. Contributors from smaller companies had a real concern that larger or competing companies could overwhelm the project and push it in a direction that would only benefit them. Over time, the largest source of contributors have changed as companies have come and gone, but by maintaining that level playing field in which all ideas are discussed independently of their source, we have been able to avoid disruptive changes in direction. We often compromise on implementation details to meet competing requirements, but that gives us more generalized solutions for problems, allowing us to meet the needs of more users, and the software is stronger as a result.
Another way we emphasize consensus is in the voting system we use to elect leaders. In the type of elections most of you are probably familiar with the candidate with the most votes will win. Often, if there is no clear winner, a secondary run-off election needs to be held. The Condorcet election system is a preference-based system. The candidate who wins the election is not necessarily the one to win the most raw votes, but the one to be ranked as a pair-wise preference over the most other candidates. In contentious elections, this can result in someone from the middle of the pack winning because they are a more acceptable second choice candidate to a larger group than any other more extreme candidate is as a first choice. There are other voting systems that use stacked ranking like this, but we chose Condorcet because of its emphasis on finding consensus candidates.
5. Set Common Goals
Besides being useful in guiding decisions, having a set of documented principles can instigate new action. For example, two of our principles, “OpenStack is built for our users” and “OpenStack First, Project Team Second, Company Third” fed into the creation of a community wide goal process for accomplishing larger initiatives that our users wanted.
This is not a top-down, command-driven system in which a central architect or project manager controls the work everyone else is doing. It is a tool for completing work by bringing focus on user needs. The system is built around two-way communication about what our user and contributor base desires and what they feel is possible, combined with a consensus model for choosing which goals are accepted.
So far we have used the goals process to reduce technical debt by moving duplicated code into shared libraries, improve deployment options by ensuring that all of our REST APIs can be deployed behind WSGI services, and port most of our applications to Python 3. We recently wrapped up a goal to make our use of access controls consistent and are actively working on a goal to provide better upgrade tools.
6. Create Decision Making Processes
Establishing the goals process was only possible in the first place because we had already defined a clear process for making decisions as a community, and we were able to build on top of that.
As a community grows larger, decisions affect more and more people and they rightly want to have input to the process. The ad hoc processes of deciding based on a quick mailing list discussion or chat session that is sufficient at the start of a project don’t always scale, so more formal processes may be needed.
OpenStack’s structure as a collection of teams gives us a quasi-hierarchy we can use for decisions based on how broad the impact is. Combining that with our principle that “contribution is our currency” means that we try to ensure that the people doing the work are the ones making the decisions. For decisions that affect the entire community, we have the Technical Committee, elected by the contributors. However, we don’t want the TC to devolve into a team of BDFLs. The members of the TC try to facilitate finding consensus, and act as a safety valve if consensus cannot be reached in another way. Sometimes just knowing that there is a separate group with the power to make decisions will motivate people to find consensus on their own.
7. Enable Others
The team structure and reliance on delegation is a reflection of our servant leadership philosophy based on enabling people rather than doing things for them.
The 13 members of the TC know we don’t scale very far, and that we don’t need to be involved in everything ourselves. We each have areas that we are passionate about and work on, but we also expect others in the community to care about different things, and therefore, we try to build systems to help them be successful. The key is to include incentives for people to volunteer, to recognize when volunteers are available, and to design systems to enable and support them. That supportive approach applies for even the most basic interactions in our community.
Code review plays a fundamental role in building OpenStack. All patches, no matter their author, must be reviewed before they are merged. Anyone can review any patch, and after demonstrating a good understanding of a project, contributors may be asked to join its core review team. Members of the core team are given permission to approve patches so they can be merged, in exchange for the understanding that they will help other contributors by reviewing submissions. They spend a significant amount of their time reviewing patches and design documents to help other contributors align their goals with the rest of the community and then achieve them.
We also place a strong emphasis on automation as another way of enabling others. For example, we have most of our testing and hosting infrastructure set up to be self-service. Anyone can submit the patch to trigger creating a git repository or release a package. Teams of knowledgeable people review those patches for technical issues, and when they are merged, the work is handled through automation. This allows anyone to start a new project and build a team without seeking permission in advance.
8. Embrace Economic Reality
We do need to pay careful attention to incentives and workload within the structures we end up building, and embrace the economic reality in which we are working.
Until recently, we had a separate documentation team made up mostly of technical writers, with some additional people helping to build tools. This structure worked well for a long time because it allowed the writers to concentrate on the interconnectedness of the various components, especially when they worked on the installation guide.
On the other hand, the writing team was always following behind the development teams, so it was difficult for them to keep up with the changes in all of the projects. This was especially true as the number of projects grew, but it was always true to some extent. The writers adapted by releasing the documentation a bit after the code, and after an intensive testing period for the installation guide — basically the writing team tried to solve the problem by having the existing members do even more work to test and revise the instructions.
Along the way, the companies employing those writers reorganized internally so that the writers were on upstream-focused teams. Then, early last year, when those companies needed to cut costs, some of those upstream teams were the first to go. We lost almost all of our dedicated writers to layoffs or reorganizations.
At first, this seemed devastating. And it was bad, not only for the community, but for the folks who had to look for new jobs. But it was just an extreme manifestation of the situation we had been in all along.
The problem with documentation being outdated or incomplete continued to grow worse over time because there was no feedback mechanism to ensure that documentation updates were made at the same time as code updates, and the documentation team was far too small to support the work of the hundreds of developers adding features and changing the software. We were hosting the documentation apart from the code because there was a separate team to own the documentation, and our governance structures encouraged that, even if the economics did not support it.
The solution was to place the documentation work near the people with the ability to ensure it was done in a timely way, create the incentive for them to do it, and then build trust to allow them to do the work. We decided to move ownership of the content from the documentation team to each of the project teams. We hoped the change of ownership would produce the incentive to keep the docs accurate, since they now reflected directly on the project team. And in fact, after this change went into effect, some of our teams created internal policies that require code changes to include the relevant documentation updates.
9. Create New Leaders
Finally, I want to talk about changes in leadership.
The OpenStack community explicitly rejected the BDFL model when the project was started. In part, this was to ensure we had a democratic system of decision making, but another benefit is that it forces us to be prepared for changes in leadership.
We did not want to build a system that relied on an individual, or a small group, for consistency over a long period of time. We knew we would need to be prepared for people to leave the project. The terms for our leadership roles are all based around development cycles, which ensures that change can happen smoothly at predictable times. Those changes are still not automatic though, because in order to have change, we need to have new people ready to lead.
The most important thing you can do as a leader is to prepare your community to go on without you by ensuring there are new leaders to take your place.
Don’t wait for people who already have leadership skills to join the project; recruit from within. Provide a path for people to find areas where they are interested in contributing so they can grow into leadership roles within your community. Having a path helps them understand what to do and how to do it within the culture you are building. It gives them the opportunity to earn the trust of the community and helps you identify the people on the path, so you can help them. You will need to provide opportunities for them to learn the roles as well as to demonstrate their abilities — especially if they will need to be elected.
Within OpenStack, we have a few leadership roles that aren’t necessarily a progressive advancement track, but that do provide different perspectives on the community and different levels and types of responsibility. For example, we frequently need to coordinate the work of the vertical project teams with horizontal teams working on things like security, release management, and QA.
A little over a year after it was founded, I was elected to lead the team responsible for producing common libraries of code that could be reused across all of the projects during a time when the number of those projects was growing rapidly. Our small team started having trouble managing our interactions with all of the other teams, and we started having some technical issues — bugs due to behavior changes — because communication broke down.
To address the problems, we created a liaison program, and asked each team consuming our libraries to designate someone to own the communication from their end. After working out a few kinks, the liaison system grew into something that worked well enough that we are now using it with all of our other horizontal project teams.
In addition to the tactical benefits of improving our communication, we have discovered a few more strategic benefits to having designated liaison roles. For example, because the responsibilities of the liaison are a subset of the team lead’s duties, the liaisons get a sense of what it’s like to manage the project team from a perspective they may not have considered otherwise. They also have an opportunity to learn from working with other teams.
Many people who volunteer as liaisons never take on a more comprehensive role as team lead. But for those who do, having the liaison experience helps them understand how to coordinate with the rest of the project and how decisions made within a project team can affect people who work in other parts of the community. They also interact with other leaders in the community more closely, which allows them to build the trust needed to function as a strong leadership team.
Our community-wide goals process is another area where our community members can gain some leadership experience beyond their primary team. We are still defining a goal champion role, taking into account the lessons we have learned from the work we’ve done over the last 6 development cycles. So far, key aspects of the role include educating teams about how and why the goal needs to be implemented, and then coordinating and tracking their work. It’s shaping up to be a bit of a cross between a senior developer and a project manager.
Acting as a goal champion provides opportunities to learn about leading our community. Champions interact with all of our project teams, and because each team is slightly different, that breadth of experience can give them good insight into how different teams work, an important aspect to understand if they are interested in taking on other community leadership roles. They also practice the communication and negotiation skills needed to coordinate work across many sub-teams simultaneously, which is no small challenge. Imagine the effort it takes to convince one open source project team to do something, and then multiply that by a couple of dozen, and you get some idea.
We try to define roles like the goal champion or liaison in a way that allows us to use them to invest in creating new leaders, rather than merely asking the same people to take on more and more work. This lets us solve new problems by having new people drive the work. It also helps our community grow, and lets people contribute in the ways they are interested.
Every new leader in the community and every division of labor means less of a chance someone will experience burn out, either from being overloaded or from doing the same task for too long. Spreading the load among more people also provides incremental steps so contributors who want to lead don’t have to leap immediately to a major elected position. They have a chance to gain experience and grow into leadership roles.
It’s not enough to create the leadership opportunities, you also have to actively recruit people to fill them. Not every capable leader in your community is going to have the confidence to jump in without prompting. Some need to be nudged a little.
I have found that a 1-on-1 personal suggestion encouraging people to take on a leadership role is the most effective approach. It gives the other person a chance to explain why they may not want to or be able to do the work, without putting them on the spot publicly. It also gives me a chance to encourage them or address any concerns or questions they have. Sometimes it takes more than one attempt, but more often than you might expect, people are interested enough that with a little encouragement they take up a new role willingly.
Every time I nudge someone to suggest they take on a leadership role, I offer my support while they are in the role. I offer to be a sounding board for their ideas, helping them to anticipate issues others might raise so they can be prepared to discuss them. Leading change can be hard, and having some input from different perspectives can make it easier. I tell them that I seek out the same sort of advice from my colleagues in the community when I have my own proposals to emphasize that they do not need to feel like they have all of the answers going into the role.
I also tell them I can provide context for discussions that started before they were involved. One of the benefits of having done this work for so long is that I was present for discussions and decisions they may not have experienced — what Hillis called the “story that starts before you can remember.”
It’s important to understand the history behind existing policies and decisions, especially in our community, where we have some long-standing issues that we haven’t been able to address to everyone’s satisfaction. Knowing the solutions that have been rejected, and why, gives new people a head start on finding the next solution to try.
As part of providing part of that historical context, I also talk to them about the ideas I’ve covered with you today.
- Think Long term
- Organize Around People
- Document Your Culture
- Build Consensus
- Set Common Goals
- Create Decision Making Processes
- Enable Others
- Embrace Economic Reality
- Create New Leaders
As you leave the conference and head back home, keep these things in mind. Plant your acorns. Help your open source communities grow strong.