Release Team Changes and Goals for OpenStack’s Mitaka Release Cycle
For the Mitaka cycle, we will be implementing changes designed to make it easier for project teams to manage their own projects, with less need for coordination and tight-coupling of the schedule.
Enabling a New Model for Milestones
During previous release cycles we have strictly synchronized teams around milestones, tagging pre-release versions of each project on or around the same day and encouraging the project team leads (PTLs) and release liaisons to ensure Launchpad reflects an accurate picture of the upcoming release, to apply feature freeze rules, etc. As the number of teams has grown, the time it takes to handle this coordination has also grown. Last cycle we delegated management of the feature freeze period, and this cycle we will be delegating the rest of that work to the release liaisons within the project teams. We will focus instead on communicating expectations clearly, and making sure that the liaisons have our support with tools and guidance.
We will also be de-emphasizing milestones as strict deadlines. Some of our projects, like Oslo, work ahead of the “standard” schedule to enable other projects to use their work, and some projects like Horizon work a bit behind the schedule because they rely on other projects finishing work before they can consume it. Both of those examples are already on slightly different milestone schedules, and we’re generalizing this for our newer projects.
We will still use a milestone-based schedule, but we will treat the milestones as reminder points rather than deadlines. We will rely on the project teams to trigger their own releases, based on our guidance and support through tools. With twelve release cycles behind us, the OpenStack community is ready to take on these responsibilities that have been focused on a small team in the past.
Stable Branch Releases
Although we wanted to drop stable branch point releases entirely, and encourage consumers of stable branches to work from the tip of the branch, we heard from distributors and deployers alike that it is hard to understand the differences between downstream packages if they are not built based on tagged releases. We discussed tagging every commit merged to the stable branch, but there are actually many patches that don’t make sense to release because they are related just to build system changes and not changes to the project code. We definitely want to stop coordinating point releases from stable branches all at the same time, so, as a compromise, we will encourage every project to tag stable branches as needed. The general rule of thumb we will use is to cut a release when a security fix is merged, and to decide on a case-by-case basis for other types of patches like bugs and performance fixes.
Communication Changes
In past cycles, we had release managers sync up with PTLs and liaisons during the weekly project meeting, or during office hours time in #openstack-relmgr-office. Finding times to meet with all of the liaisons proved challenging, especially since our office hours did not overlap well with some time zones.
This cycle, we will be focusing on using the openstack-dev mailing list for communication. All official plans and announcements will be made via email messages with the “[release]” topic in the subject line. Everyone is encouraged to post questions on the list using the same topic, so answers are easily visible to the entire community.
To supplement the individual announcement threads, we will send “release countdown” messages as important dates approach. These messages will contain schedule reminders, dates, and other information about the things liaisons and project teams should be focusing on around the time the email is sent.
Since we are no longer using #openstack-relmgr-office for office hours, we have renamed the channel to #openstack-release. Most members of the team have a bouncer configured to stay in that channel, so questions there will be found when we come online.
Finish Release Automation
We introduced the openstack/releases repository during the Liberty cycle. Project teams submit patches to the repository to request new release tags. For Liberty, the release team processed those requests manually, running the scripts that we had used in previous release cycles. For Mitaka, our major goal is to finish the work to automate the release process so we no longer need to run release scripts by hand. We will be working closely with the Infrastructure team to set up the necessary tools and safeguards to allow us to implement hands-off automated releases and milestone tagging.
Simplify Change Management Tracking
This work of aligning the view of the world in Launchpad with the actual changes happening in project repositories is one of the most time consuming aspects of release management. Marking the bugs as closed in the correct release and updating the blueprint status fields has been taking more and more time, as the number of projects and the number of changes within projects has continued to grow. Part of the reason it is so much work is we are using the same tool for contributor planning and for tracking the content of each release (change management) within the release process. To solve the problem, we are moving release content tracking out of Launchpad, to use a new tool. This leaves Launchpad for contributor planning and scheduling, but since it isn’t tied to the release process project teams can adopt their own variations in how they use Launchpad, giving them more flexibility and autonomy.
To replace the features of Launchpad that were used for change management, we have build a couple of new tools.
First, besides the data files with the release history of all of our projects, the releases repository also includes code to render an HTML version of that data, letting us build a release history site across all projects and all release series.
The other tool, called reno, is used within each project source tree to manage release notes. Rather than having to update bug or blueprint records separately when patches are merged, as we do now, reno lets us add release notes right in the source code repository with the project. Adding documentation to a project repository is easy enough with any simple text file, but reno works using individual files for each note to make them easier to review and manage across multiple branches. By ensuring each file has a unique name, reno will let us carry release notes into stable branches when fixes are back-ported, without introducing any merge conflicts.
reno is being rolled out to projects right now for Liberty stable releases and for the Mitaka cycle release. When projects have adopted it, and their notes are being published, we will add links to http://docs.openstack.org/releases to make it easy to find the release notes for each project.
Summit Talk
Because these changes to our release process affect all OpenStack projects, Thierry Carrez and I have been working to publicize them in the community. We spoke in Tokyo, and the video and slides are available online.