git-os-job 1.1.1

The OpenStack project stores the logs for all of the test jobs related to a commit on http://logs.openstack.org organized by the commit hash. To review the logs after a job runs, most developers start with the message jenkins leaves on gerrit, and click through to the log files. Not all jenkins jobs are triggered by or related to a gerrit review, though (e.g, release tags).

git-os-job makes it easy to find those logs by finding the hash of the commit and using it to build the right URL. It will then either print the URL or open a web browser directly.

What’s new in 1.1.1?

  • don’t decode bytes to unicode twice

git-os-job 1.1.0

The OpenStack project stores the logs for all of the test jobs related to a commit on http://logs.openstack.org organized by the commit hash. To review the logs after a job runs, most developers start with the message jenkins leaves on gerrit, and click through to the log files. Not all jenkins jobs are triggered by or related to a gerrit review, though (e.g, release tags).

git-os-job makes it easy to find those logs by finding the hash of the commit and using it to build the right URL. It will then either print the URL or open a web browser directly.

What’s new in 1.1.0?

  • add –reverse option to go from log URL to review URL
  • decode command output for python 3 support
  • add -u alias for –url command

Stop Working So Hard: Scaling Open Source Community Practices

Lately, I have been revising some of the OpenStack community’s processes to make them more sustainable. As we grew over the last 7 years to have more than 2,000 individual contributors to the current release, some practices that worked when they were implemented have begun causing trouble for us now that our community is changing in different ways. My goal in reviewing those practices is to find ways to eliminate the challenges.

OpenStack is developed by a collection of project teams, most of which focus on a feature-related area, such as block storage or networking. The areas where we have most needed to change intersect with all of those teams, such as release management and documentation. Although the teams responsible for those tasks have tended to be small, their members have been active and dedicated. At times that dedication has masked the near-heroic level of effort they were making to keep up with the work load.

When someone is overloaded in a corporate environment, where tasks are assigned and the performance and workload of team members are reviewed regularly, the employee can appeal to management for help. The solution may be to hire or assign new contributors, change the project schedule, or to make a short term trade-off that incurs technical debt. However, open source projects are largely driven by volunteers, so assigning people to work on a task isn’t an option. Even in a sponsor-driven community such as OpenStack, where many contributors are being paid to work on the project overall, sponsors typically give a relatively narrow mandate for the way their contributors can spend their time. Changing the project schedule is always an option, but if there are no volunteers for a task today, there is no guarantee volunteers will appear tomorrow, so it may not help.

We must use a different approach to eliminate the need for heroic effort.

Continue reading “Stop Working So Hard: Scaling Open Source Community Practices”

Lessons learned from working on large scale, cross-project initiatives in OpenStack

I have been involved with OpenStack development since just before the Folsom summit in 2012. Over the course of that time, I have participated in innumerable discussions about 3 big features tied to OpenStack logging: translating log messages, adding request IDs to log messages, and adding unique message IDs to log messages. We have had various degrees of success with the design, implementation, and ongoing maintenance of all three features, and the reasons for success or failure in each case provide helpful insight into how to approach changes with large community and product scope that should be considered before our next discussion at the summit/forum in Boston in 2017.
Continue reading “Lessons learned from working on large scale, cross-project initiatives in OpenStack”

Driving OpenStack via Ansible

Last week I spoke at the Atlanta OpenStack meetup about “Driving OpenStack via Ansible,” in which I introduced Ansible as a tool and talked about its ability to integrate with OpenStack. As part of the presentation I used two playbooks to launch VMs on a cloud and configure them with different applications. We walked through the playbooks and talked about what they were doing, the things that tripped me up while writing them, and then brainstormed ways to use Ansible in situations that have come up for members of the meetup.

One playbook uses my role to install ZNC, the popular IRC “bouncer,” for maintaining a persistent chat presence. The other demo was based on a playbook with the roles needed to configure a server for OpenStack development, ready to run devstack.

The slides are available, and you can download the playbooks from the github repository and try them yourself.

We used Dreamhost’s public cloud, DreamCompute, for the demo at the meetup. Thanks to the DreamHost crew for providing those resources!

Continue reading “Driving OpenStack via Ansible”

How OpenStack Makes Python Better, and Vice-Versa

I’ve been a Python developer since the late 1990s and came to the OpenStack project from that long background within the rest of the community. Thierry Carrez is on the staff of the OpenStack Foundation and the chair of the OpenStack Technical Committee. He came to Python through OpenStack’s adoption of the language.

At EuroPython 2016, we delivered a presentation titled “How OpenStack Makes Python Better, and Vice-Versa”, discussing the history of the decision to use Python, our perspectives on how the two communities benefit each other, and how we can improve the relationship. The recording of the presentation, and the slides, are below.

Continue reading “How OpenStack Makes Python Better, and Vice-Versa”

OpenStack contributions to other open source projects

As part of preparing for the talk I will be giving with Thierry Carrez at EuroPython 2016 next month, I wanted to put together a list of some of the projects members of the OpenStack community contribute to outside of things we think of as being part of OpenStack itself. I started by brainstorming myself, but I also asked the community to help me out. I limited my query to projects that somehow touched OpenStack, since what I am trying to establish is that OpenStack contributors identify needs we have, and do the work “upstream” in other projects where appropriate.

OpenStack has many facets, and as a result has pulled in contributors from many parts of the industry. A large number of them are also members of other open source communities, so it’s no surprise that even with only a few respondents to my question (most of them privately, off-list) we came up with a reasonably long list of other projects where we’ve made contributions. I did not make a distinction between the types of contributions, so this list includes everything from bug reports and triage to documentation to code patches for bug fixes or new features. In several cases, the projects came into existence entirely driven by OpenStack’s needs but have found wide adoption outside of our immediate community.

Python Packaging

  • packaging
  • pip
  • setuptools
  • wheel

Python Web Tools

  • Pecan
  • requests
  • WebOb
  • Werkzeug
  • wsgi-intercept
  • WSME

Python Database and Data tools

  • alembic
  • python-memcache
  • Ming
  • Pandas
  • redis-py
  • SQLAlchemy

Python Testing

  • fixtures
  • testtools
  • testrepository
  • tox

Other Python libs and tools

  • APScheduler
  • dogpile
  • eventlet
  • iso8601
  • jaraco.itertools
  • ldappool
  • Mako
  • pykerberos
  • pysaml2
  • retrying
  • sphinxcontrib-datatemplates
  • six

Python Interpreters

  • CPython
  • PyPy (in the past)

Messaging

  • kazoo
  • kombu
  • pyngus
  • qpid
  • RabbitMQ

JavaScript

  • AngularJS
  • Registry-static
  • “other JS libraries”

Deployment, Automation, and Orchestration Tools

  • Ansible
  • Ansible modules for OpenStack
  • Puppet & Puppet Modules
  • Chef modules for OpenStack
  • saltstack

Linux

  • cloud-init
  • dpkg
  • libosinfo
  • Linux kernel
  • LUKS disk encryption
  • systemd

Virtualization

  • kvm
  • libguestfs
  • libvirt
  • qemu

Networking

  • Dibbler (DHCP)
  • OVS
  • OpenDaylight

Containers

  • Docker
  • Kubernetes
  • openvz

Testing and Developer Tools

  • gabbi
  • gerrit
  • Zuul
  • Jenkins Job Builder

Cloud Tools

  • fog
  • libcloud
  • nodepool
  • owncloud
  • phpopencloud
  • pkgcloud

Linux Distributions

  • Ubuntu
  • Red Hat
  • Debian
  • Fedora
  • Gentoo
  • SuSE

Other Tools

  • caimito (WebDAV front-end for object storage)
  • Corosync (cluster & HA synchronization)
  • Etherpad-lite
  • greenlet
  • jarco-tools
  • MySQL
  • Zanata (translation tools)

Updated 23 June to add Kubernetes to the list of container projects.

Updated 24 June to add pysaml2 to the list of Python libraries.

OpenStack Mitaka Release Complete

I’m excited to announce the final releases for the components of OpenStack Mitaka, which conclude the 6-month Mitaka development cycle.

You will find a complete list of all components, their latest versions, and links to individual project release notes documents listed on the new release site.

Congratulations to all of the teams who have contributed to this release!

I also want to extend a big thank you to Ajaeger and jhesketh for their help with build issues today, and to ttx and dims for their help with release work all this cycle and especially over the last couple of weeks. 

Thank you!

Our next production cycle, Newton, has already started. We will meet in Austin April 25-29 at the Newton Design Summit to plan the work for the upcoming cycle. I hope to see you there!

get