OpenStack Release Management Changes for Mitaka Retrospective
The OpenStack Release Management team has been focusing a lot on automation during the Mitaka release cycle. At the start of the cycle we set out our goals, and we’re making good progress toward completing them. We’ve made quite a few supporting changes to the release process, with considerable help from the Infrastructure team.
Standard Release Models
We have standardized projects on one of three different release “models”, describing the frequency of releases. The models are applied to projects using tags in the governance documentation, to communicate that information with consumers of the projects. The release:cycle-with-milestones model is the one most folks are used to, since it is the model most projects have had until fairly recently. It involves projects preparing pre-release deliverables at set times during the cycle, with a final full release at the end. The release:cycle-with-intermediary model allows projects to release full releases at any point in the cycle, with a final at the end forming the basis of a stable branch. Most of our libraries use this one, and we have a few server projects like Ironic and Swift following it, too. The release:independent model is for projects not tied to the release cycle. Independent projects may hold tools used while working on OpenStack but that are not part of a production deployment, or newer projects that have not yet synced their development with the release cycle. These projects typically don’t have a stable branch for a release series (though they may have other stable branches) and they don’t follow the deadlines.
Standard Versioning Scheme
We have also standardized all version numbers so that we use “semantic versioning.” Semantic versioning allows someone comparing two version numbers for the same project to tell the relative difference in the different releases. All of our versions include three components, the major, minor, and patch levels. When a new release includes only bug fixes, the patch level is incremented. That indicates that the new package should be compatible with the previous release, and can be dropped in as a replacement safely. When a release includes new features or changes in dependencies, the minor version number is incremented. These new packages should also be compatible, but may require updating other components. And when a new release is known to be incompatible, the major version number is incremented. Many of our server projects will do this each cycle to indicate that upgrade work is expected, and libraries do it when incompatible API changes are made.
Communication Changes
We have made some important changes to the way we communicate information about releases. The release management team no longer maintains milestone series pages for each project on Launchpad. Instead, we have created http://releases.openstack.org, and link to the release artifacts there. This new site makes it easier for consumers of OpenStack projects to find all of the related releases for a given series, and is easier to update with our new automation. Most project teams are still using Launchpad for their planning, but many are not and it is no longer required. Bug reports for projects should still be filed through Launchpad.
Reviewable Release Requests
The releases site is built from the same data files that power the new automated release process. We’re not quite 100% automated, so we still have a couple of manual steps, but the entire process is set up to support reviews and to allow anyone on the release team to process a release request from any project. Release liaisons or PTLs for a project can submit a patch to the openstack/releases repository describing the new release they want. The release team then reviews that patch, helping them to ensure they are using the semantic versioning rules correctly, that they are releasing all of the changes they want to include in the new release, and that the timing for the release is “good” (we try not to release new versions of libraries late in the week, for example). When the request is approved, a new release artifact is created, and published to tarballs.openstack.org (and pypi.python.org for Python libraries).
The release process works the same for all branches, and is simple enough that we are now releasing stable branch updates more often than in the past. That should make it more appealing for downstream consumers to recommend fixes to be back-ported (and, we hope, submit the patches to do so), because they will see a new release including that fix more quickly than in previous cycles where we waited for pre-arranged milestones to create new releases.
Release Notes
The new releases web site also includes links to the release notes for projects. We are still updating the site to link to the new Mitaka release notes pages, but the links to the Liberty notes are in place. Most projects are using a new tool to manage the release notes in the source tree with the other parts of the project, instead of separately in the wiki. The new tool, reno, makes it easy for contributors to write notes as they fix bugs or add features, which means we should have more complete and detailed release notes by the end of the cycle. Because the notes are with the source code, they are also copied into branches when fixes are back-ported, so the release notes for stable releases will be updated automatically.
Dependency Management
Aside from the release automation, the team has also been working on managing the way we handle dependencies for projects, to make our CI systems more reliable. In the past we have had issues introduced when new versions of packages were released (sometimes even our own). A breaking change to an API, or even a minor bug in a library, would cause all sorts of test jobs to fail, blocking the work of other contributors until a fix was prepared. The new system being rolled out uses a set of constraints to indicate exactly which versions of packages should be used in the test jobs. Each project still declares compatibility with a range of each of its dependency, allowing flexibility for deployers and packagers. But in our upstream test jobs, the constraints reduce the impact of new releases on our test jobs, and give us a way to verify that a new version works before adding it to the test system. Constraints are already in place for the integration tests, and the tools are there for projects to add them for unit tests as well. We have a few early adopters who have set that up, and we’ll be encouraging other teams to do so during the Newton cycle.
Looking Ahead to Newton
We didn’t finish everything we set out to do in Mitaka, so some of that work will carry over to our next cycle. We need to finish the release automation by working with the Infrastructure team so that approving a release request in gerrit triggers the release and there are no more manual steps. When that work is complete we plan to expand the release automation’s use beyond projects with the release:managed tag to be used by default for all official projects. We also plan to implement translations for release notes, an important feature we had for past releases but that did not make it into this release.