So, you’ve released a broken package to PyPI. What do you do now?
Step One: Relax. It happens.
Step Two: Realize that just like with email, there is no way to recall a message once it has been sent.
Step Three: Release a new, fixed, package.
Recently, several popular and important Python projects have had issues with releases. The nature and scope of the issues varied, and it’s not necessarily constructive to point fingers. But the way the issues were dealt with, by deleting packages from PyPI, ended up causing a cascade of further problems that the package maintainers seem unaware of. So we are writing today to highlight the best practices for dealing with bad packages after a release has already gone public, with the hope that repeated occurrances can be avoided.
There are, as described above, three steps to dealing with a bad package. The first two steps are essential. You’ve broken something, and it can be embarrassing and frustrating. People may notice. The more popular your package, the more likely it is they will, but it happens to everyone. Depending on how broken things really are, you may feel a lot of pressure to fix it fast. Bad packages are easy enough to fix, but staying calm in the face of problems resulting from mistakes is important for finding the right way to fix them and helps to prevent reactionary steps that break even more things.
After a package is published to PyPI, it is out in the wild. It exists in caching servers, proxies, and wheelhouses, mostly in places you can’t reach to delete it. Removing the package from PyPI seems like a natural solution, but it actually makes the problem worse instead of better. Removing the package from PyPI will not help anyone running a large CI system with a significant package cache. It won’t help anyone doing continuous deployment to production from such a cache. All of those folks will still have the broken package, and will have the further challenge of debugging where in the world it came from if it’s no longer available on PyPI. Your tweets and IRC messages about removing the package will go unnoticed by folks using your software, but not following you on social networking channels.
Accepting that you cannot turn back time or effectively revoke the bad package keeps your choices about what to do next mercifully limited to thinking about that third step: how to roll people forward out of the current place they may be in.
At the source code level, if there is not an immediate simple fix, the
best response is often to revert the broken change. Do this as a new
revert commit, on top of the current state of the branch (history only
moves in one direction). Once you’ve done that and have verified that
it fixes the breakage, it’s time to publish a new release. You want to
use an incremented patch version for the new release, rather than
reusing an existing number. By rolling the version forward you get the
fix to anyone who has a configuration management system that does the
equivalent of pip install -U your-package. It also means that anyone
running pip with uncapped requirements on your package will get the
new, fixed, version instead of the old, broken, version.