So, you’ve released a broken package to PyPI. What do you do now?

Step One: Relax. It happens.
Step Two: Realize that just like with email, there is no way to recall a message once it has been sent.
Step Three: Release a new, fixed, package.

Recently, several popular and important Python projects have had issues with releases. The nature and scope of the issues varied, and it’s not necessarily constructive to point fingers. But the way the issues were dealt with, by deleting packages from PyPI, ended up causing a cascade of further problems that the package maintainers seem unaware of. So we are writing today to highlight the best practices for dealing with bad packages after a release has already gone public, with the hope that repeated occurrances can be avoided.

There are, as described above, three steps to dealing with a bad package. The first two steps are essential. You’ve broken something, and it can be embarrassing and frustrating. People may notice. The more popular your package, the more likely it is they will, but it happens to everyone. Depending on how broken things really are, you may feel a lot of pressure to fix it fast. Bad packages are easy enough to fix, but staying calm in the face of problems resulting from mistakes is important for finding the right way to fix them and helps to prevent reactionary steps that break even more things.

After a package is published to PyPI, it is out in the wild. It exists in caching servers, proxies, and wheelhouses, mostly in places you can’t reach to delete it. Removing the package from PyPI seems like a natural solution, but it actually makes the problem worse instead of better. Removing the package from PyPI will not help anyone running a large CI system with a significant package cache. It won’t help anyone doing continuous deployment to production from such a cache. All of those folks will still have the broken package, and will have the further challenge of debugging where in the world it came from if it’s no longer available on PyPI. Your tweets and IRC messages about removing the package will go unnoticed by folks using your software, but not following you on social networking channels.

Accepting that you cannot turn back time or effectively revoke the bad package keeps your choices about what to do next mercifully limited to thinking about that third step: how to roll people forward out of the current place they may be in.

At the source code level, if there is not an immediate simple fix, the best response is often to revert the broken change. Do this as a new revert commit, on top of the current state of the branch (history only moves in one direction). Once you’ve done that and have verified that it fixes the breakage, it’s time to publish a new release. You want to use an incremented patch version for the new release, rather than reusing an existing number. By rolling the version forward you get the fix to anyone who has a configuration management system that does the equivalent of “pip install -U your-package”. It also means that anyone running pip with uncapped requirements on your package will get the new, fixed, version instead of the old, broken, version.

This post was co-written with Monty Taylor.

  • MinchinWeb

    Once you’ve released the updated version to PyPI, is there any reason to keep the broken package up there? Or would best practice be to remove it, now that the updated and working version has been released?

    • Nick Coghlan

      No more so than there is with any other release with known bugs – folks that happened to pin the broken version (since the error didn’t affect them) will appreciate it remaining available, while everyone else will automatically upgrade to the new version.

      • Doug Hellmann

        Nick is exactly right. There’s no reason to remove the package. Leaving it in place allows folks who are pinning their requirements to upgrade when they are ready, and the new package will automatically be selected by anyone who is using uncapped requirements.

        • Vladimir Rusinov

          There is at least one good reason – security issues. Packages with known security bugs should be removed, no matter how “good” they work.

          • Doug Hellmann

            I go back and forth on this case. I tend to think they should not be removed, ever, but i do see the point. However, if they *are* removed, that should only happen after an updated package is available.

          • Giovanni Cannata

            In this case you could deploy a “post” package with the security fix. It overwrite the unsecure package and is downloaded by pip even if the original one was requested. For example if you release v1.2.3 and then find a security hole in it you can release v1.2.3.post1 and it will override v1.2.3. Even if pip requests v1.2.3 pypi should return v1.2.3.post1.