Notes from OpenStack Folsom Design Summit Spring 2012

Originally posted · 14 min read

These are my notes from the Design Summit and Conference kicking off the Folsom release of OpenStack, held April 16-20, 2012 in San Francisco, CA. Much more happened than I could hope to capture in a single post, but I can publish most of my notes from the design sessions.

tl;dr

This was a good week for the DreamHost development team. We began establishing ourselves as contributors in the OpenStack community, met a lot of people we will need to be working with over the next few months, and learned a lot and OpenStack in general. We also identified several projects that need our help, and found some tools we should be able to use for our implementation.

Goals for the Week

I had several goals for this trip.

  • Because this was my first summit, the most important thing for me to do was introduce myself to, and become acquainted with, members of the community. This will be ongoing work, but I made a good start.
  • DreamHost is making it a priority to contribute to general technical discussions, not just the aspects of OpenStack that will affect us directly (they all do, in the long run). I participated in several of the summit sessions, sharing my own thoughts, expressing requirements for DreamHost, and repeating information discussed in other sessions. (see notes below)
  • I also wanted to identify projects to which I or DreamHost can and should contribute early.

Session Notes

Dough: Billing Service

Zhongyue Luo of Sina Corporation presented “Dough,” the billing service they have created. There was a lot of discussion, some of it quite critical. The main issue raised was that Dough requires changing Horizon to add explicit hooks to call into the new service, which is brittle in the face of failed operations (such as when an instance creation is requested but cannot be completed). It also combined metering and billing too tightly and did not actually track compute or network resource usage.

DreamHost needs a metering service, and may want some features of an existing billing service to coalesce metered data before feeding it into our charge-back system.

See also

Keystone Secret Storage

Justin Santa Barbara discussed a system for using chained secrets to allow encryption of data within Swift, without requiring re-encrypting that data when the user changes their password (the first step of the chain). The idea is to create a second secret value associated with the user and encrypt it using the password, then store it in Keystone (or some system to which Keystone delegates). The chain of trust would be the user password, a generated user key, a generated tenant key, and a container key. The weak link is still the password, but using a chain allows some of the values to be changed without re-encrypting everything in the database and offers some compartmentalization for protection in the case of a breach.

OpenStack Common

Mark McLoughlin led this session reviewing the progress on the common library project to unify the implementations of features within OpenStack where it makes sense to share code.

The plan for now is to provide a two-stage system for common code. Incubated code would live in a special part of the common repository and would be copied into projects that use it. When the API for an incubated module stabilizes, the module can be moved into the common library as an official module and it can be imported from that namespace instead of the project namespace. That deprecation will happen over the course of several releases (deprecate in N, keep support in N+1, remove in N+2) to give downstream users of the project code trees time to adjust. This process will begin with the config module during the Folsom release cycle.

Existing APIs to be considered for common include:

  • config (cfg, pastedeploy, logging)
  • setup
  • importutils
  • authutils
  • timeutils
  • local – greenthread local storage
  • context
  • exception
  • wsgi, extensions (Nova)

The two existing developers on the project asked for help reviewing patches and contributing. Since code cleanup is a priority for DreamHost, I have added this to my todo list.

See also

Standardizing Database Management

This session as a grab-bag of topics related to database management issues.

Database management works slightly differently between the projects. Specifically, handling dropped connections for MySQL needs work. Someone suggested contributing code back to sqlalchemy to help with that.

Database migrations with sqlalchemy-migrate can be painful. Jonathan LaCour suggested using Alembic instead.

Someone else proposed a rule that database migrations and code changes should be submitted in separate changesets, sequenced so that nothing breaks. There was some discussion, but I don’t think we agreed to do this.

We did agree to disable automatic database creation and migration for all projects.

See also

Stable Branch Management

Mark McLoughlin led this session on stable branches and support for historical versions. There was quite a bit of discussion, and the outcome looks good for users such as DreamHost.

First, representatives for several distros were in the room and they all agreed that the original proposal of dropping support for a release after 1 year was not going to work. They committed resources to the stable branch management team to improve that situation, and the core developers agreed to allow support for a given release to be advertised so long as there are actual active developers maintaining it.

The maintenance model will have two states: The project will receive “active” maintenance by the stable team during development and the next release cycle. It will then receive “passive” maintenance (only security and high-impact fixes) as long as there are developers signed up to provide it. The maintenance status of any given release will be published (probably in the wiki) so deployers can make informed decisions.

In all cases, patches should be applied to trunk before being back-ported, and no back-port change will be approved without a reference to the trunk changeset.

Mark also discussed a tool called “palaver” for reviewing git changesets in master and tagging them for cherry-picking to be merged into stable branches. There was some discussion of integrating the features with gerrit to make triaging easier.

See also

Unified Command Line Interface

Dean Troyer of RCB has started working on a design for a unified command line interface to replace all of the existing clients. The project goal is to provide a consistent user interface with naming conventions, argument names, etc. For Folsom we will develop a prototype and work out the arguments supported by each command. Dean has done a fair bit of analysis of the existing commands to map them to a new structure he is proposing, and he and I are both creating prototypes on which to base the design discussions.

We plan to start a new project to hold the base classes and core components of the CLI, then provide a plugin system to allow other projects to define their own sub-commands. The commands available on a given system will depend on the packages installed.

There was also some discussion of sharing more code between the Python client libraries for the various projects, which would make implementing the new CLI easier. Those changes do not block work on the CLI, so the projects can continue in parallel.

Dean and I had several conversations after the session about how to implement the plugin loading. We both reviewed cement2, a library identified by some of the Quantum developers, and found it unsuitable (I categorized it as the unholy love child of a Zope developer and a Java developer, but that may be a bit harsh). Dean posted his prototype, and I have started a new framework called cliff that I am proposing.

See also

Splitting up Nova

Thierry Carrez led a session on looking for other parts of Nova that can be split out into their own projects to make maintenance easier. Suggestions included volumes (already approved), hypervisor drivers, utils (into openstack-common), the job/resource scheduler, RPC and notifiers (also into openstack-common). Blocking issues identified included database migrations and the lax API with virtualization drivers.

The objective during the Folsom release cycle is to create a plugin system to allow pieces of code coming from these other projects to be brought into Nova at runtime when needed.

Important aspects of the plugin system are that plugin code needs to be able to execute synchronously or asynchronously, plugins need the option of two way communication (i.e., having Nova ask the scheduler a question and wait for the answer), and error handling needs to support aborting operations.

Andrew Bogott and I had a lengthy discussion about the plugin API. Our approaches are a little different (Andrew has proposed a monolithic plugin API which will have entry points for specific purposes and I have been arguing in favor of more targeted plugin APIs where the code that uses the plugin is responsible for loading it). We have some work to do before we agree, but we have agreed that we need to agree and will eventually, so the current disagreement is at least agreeable.

Eric Windisch suggested that all communication with the plugins should be via RPC. Neither Andrew nor I thought that should be a hard requirement.

See also

Making Nova Tenant Data “Pluggable”

Phil Day from HP led this session on changing Keystone to provide a way to store generic user data such as SSH keys, quotas, and security group definitions. The initial proposal discussed those items explicitly, but I suggested making a more generic key/value store API and that was received well by several participants.

It was unclear at the end of the session how Phil would be proceeding, so I will need to keep an eye on the mailing list for discussions about this.

See also

http://etherpad.openstack.org/FolsomPluggableTenantData

DevStackPy

Joshua Harlow of Yahoo! presented the work we have done on the replacement for devstack. Although there was some interest, there were some requirements met by devstack that are not met by devstackpy. The primary objection seems to be the fact that the documentation team does actually read and pick apart the shell script in order to create the installation guides, and that would be more difficult with the Python-based implementation. This was a little disappointing, since DreamHost had contributed persona support and some significant rearchitecting to DevStackPy to make it easier to add new platform support.

See also

Nova Orchestration

This session discussed “orchestration” in somewhat generic terms. There is a project with a code-name, but that wasn’t the subject.

There was a good bit of back-and-forth about whether to use a state machine or workflow model for orchestration. The general consensus was that a state machine combined with idempotent task implementations would allow for more reliable retry and cleanup, so it was preferred.

Transactional consistency across services is not a goal because it leads to brittle solutions.

The discussions were focused around the sub-tasks for things like creating instances in Nova, but implementation will need to extend to Glance and Quantum in the future, since they also have complicated multi-step operations. Managing parallel jobs such as creating multiple instances was tabled for future work.

The first step for now is to add a way to store the “expected” state of the system so the current state can be compared against it.

See also

Common Image Metadata

Glance provides a way to store custom properties with images. Justin Santa Barbara would like to develop a standard naming scheme for these so we can add useful metadata in a common way without requiring code changes to add those fields to all images. The specific fields Justin wants to add are mostly for identifying images programmatically, so they include things like the image distro, vendor, and version. He also has a date field for recording how up to date the image is in case it is not a base image.

There was a bit of bike-shedding on syntax and naming, but I think in the end we reached consensus on using a prefix of “org.openstack__” for common properties. Other providers who define properties would use their own prefix. Property names should have version numbers (so, “prefix__1”) in case the semantics of the field change.

Distributions will be identified by a registered name such as “com.ubuntu,” optionally followed by a qualifier or distribution name. For example, “com.ubuntu.release” and “com.ubuntu.daily” might both have the same version information but because the daily builds are not tested as thoroughly Canonical wants to make sure they are clearly identified.

I proposed using the standard in PEP 396 for version string format instead of inventing a new convention.

I also proposed investigating other standards such as DMTF and OVF for properties we might be leaving out. No one had looked at those, yet.

See also

Efficient Metering

Nick Barcet proposed some requirements for a metering system. He identified three parts to a full billing system: metering, analysis, and billing. This session was limited to the metering part, since it needs to be built first.

The proposal he has put forward includes agents collecting data for “counters” on a regular basis (hourly), and the data being aggregated at larger time periods until it is eventually discarded.

Someone brought up monitoring, which is related but would need more timely and fine-grain data.

James Peinck from Yahoo! has started implementing data collection agents which are not yet open but will be. He is interested in adapting it to work with the standard.

See also

Nova Ops Pain Points

Michael Still of Canonical led this session during which many members of the DevOps (or regular Ops) community described various issues they have encountered running OpenStack. The big issues are around upgrades and debugging, and there were some good suggestions for short and long term solutions. There was a definite consensus that operators should not need to access the database to debug or clean up an operation, so we should keep that in mind as we design the metering solution and other new features.

See also

Federated Zones

Christopher MacGown of Piston Cloud led this session talking about how they handle the lack of zone awareness in clients (Keystone can return a list of endpoints but all of the current clients only take the first one). They use a load balancer so they only need one “end point” and the work can be split out by the LB. Ken Pepple of Internap said they are doing something very similar in their implementation.

The proposal was to add zone awareness to the client so it can make decisions about where to run tasks based on geography, permission, capability, etc. (depending on how zones are defined in a given implementation). This would work by having the end-user specify an explicit reference to some object such as a Swift container. The client would determine the zone where the object lives, and use that zone for the current task.

There was some discussion of having the zone awareness be more “hint based” and allow the system to make alternative choices, but that sort of guessing is really hard to get right so we decided that if the operation cannot be performed with that zone for some reason, the job should fail.

In order to achieve this, the Keystone API needs to be changed to include the zone information associated with each endpoint that it returns in the service catalog.

See also

Making Configuration Easier

Dan of Puppet Labs talked about some of the issues he has had modeling the OpenStack configuration features to implement deployment orchestration in Puppet. Two key issues are the paste.ini file in Keystone (because it combines code and local configuration) and the monolithic Nova configuration file.

I pointed out that in the Python 3 session Paste was brought up as a blocker for moving to Python 3 support, so the configuration question was another reason to consider dropping it.

Someone else mentioned that users/installers do actually need to change the pipeline in the paste.ini, so we can’t just put that “in code” somewhere.

Quantum CLI Rewrite

Jason proposed using cement2 to rebuild the Quantum command line interface. Some requirements for the new tool are that it should be able to figure out which extensions are installed on a given server so it can pre-validate requests (or possibly turn off features in the client). The output also needs to be more parsable so the tools can be used in scripts.

This work will probably wait for the unified CLI project to establish the proper framework before continuing.

See also

DevOps “Love-In”

Duncan McGreggor of DreamHost led this session trying to get the DevOps community to combine efforts to raise awareness of their issues. The general theme that I took away from the session was that the operators need to be the squeaky wheel and commit resources to opening and tracking bugs. They can also help by triaging and commenting on existing bugs, discussing issues on the main mailing list, and providing environments where devs can recreate problems that only happen “at scale.” Duncan is going to lead an effort to summarize ops issues for the weekly project status meetings on IRC.

See also