Removing Bitrot From Virtualenvwrapper's Packaging, with a little help from my friends

After moving virtualenvwrapper to GitHub, the next phase of updates was for the packaging. Until this year, the last release of virtualenvwrapper was 4.8.4 in February of 2019. In some ways, a lot has changed in the Python packaging ecosystem since then. In other ways, not so much. As I tried to update the project, things got a little messy, but I think we’re all set, now.

History

The OpenStack community created Python Build Reasonableness (pbr) to standardize the packaging automation for the dozens of deliverables they produce every release cycle. Pbr uses entry points and other hooks to tie into different phases of setuptools build process to do things like manage version numbers, produce a changelog and release notes, build the documentation for a project, and build a list of the files that need to be included in the package. By using pbr, application and library authors can focus on writing and testing their code, and leave packaging and delivery up to the release management team. The owner just has to tag the project, and everything else is handled as part of the automation pipeline.

I like the level of automation pbr introduces, and have used it for all of my projects since it stabilized. Recently, though, I started seeing deprecation warnings when building packages. It seems that some of the integration points are triggering imports of deprecated setuptools modules like easy_install. I could try to fix that, but I thought I would take a look at the newer options for packaging. It’s been a bit of a journey.

Confusion mounts

I have been working on Kubernetes and building tools in Go for the last few years, so although I still have an interest in the Python ecosystem, I have not been following it that closely. My impression was that setuptools as a front-end for producing artifacts with commands like python setup.py sdist was deprecated in favor of other tools, so I was especially interested in understanding the state of the art there. I started by looking at the Python Packaging User Guide for the latest advice. Although maintained by the Packaging Authority, that site seems to be a collection of old and new advice, so figuring out which pages were relevant wasn’t easy.

The first guide I found by searching for “packaging python projects” still explained how to use setup.py, for example. The tutorial Packaging Python Projects, on the other hand, mentions the newer pyproject.toml standard. The tool recommendations page suggested using setuptools to manage metadata, and build to produce artifacts. There is also a list of “key projects”, which seems to list every available alternative in each category.

Sometimes venting on social media does help

I posted about my confusion on Mastodon

takes a deep breath

I have recently started seeing deprecation warnings when I build packages of my Python projects. I want to update to more modern tools. What should I be using?

I have been sending packages of Python code out onto the internet for more than 25 years now (I’m fairly sure I uploaded stuff to Usenet before distutils was a thing), and I honestly can’t figure out what the state of the art for packaging is today.

packaging.python.org/en/latest/flow/ mentions 7 different tools I can use to process my pyproject.toml file. How do I choose?

There’s reference to a (new?) tool called just “build”, whose docs recommend I install it by checking the source out from git? https://pypa-build.readthedocs.io/en/stable/installation.html

I feel completely lost trying to contribute to this community any more. Please, someone tell me I’m just bad at searching for instructions.

And, lo, they did.

I had a few people express sympathy for me, and new developers coming to the problem for the first time.

Brian Okken recommended flit as easy to use for pure Python packages. That recommendation was later seconded by Juan Luis. Unfortunately in this case, virtualenvwrapper is not pure Python, so I suspected I would have some issues if flit is focused on that use case.

Matthew Martin recommended using pyproject.toml and then choosing the right tool for each of the steps of installing, building packages, and then uploading the packages. At that point, I had not yet found the reference for what pyproject.toml should include, but this general advice matched what I expected. The main tool I needed was for creating packages, and Matthew recommended Poetry. A quick look at the web site made me think it was more of its own ecosystem than a standard tool. That may not be accurate, but it was my impression.

I thought if the Packaging Authority was recommending build, I should give that a closer look. The installation instructions were a little odd, though, coming from a tool managed by people creating tools for building packages:

The recommended way is to checkout the git tags, as they are PGP signed with one of the following keys…

That later proved to be a distraction, as build can be installed by pip.

Finally, David Beazley shared a link to Dane Hillard’s book, Publishing Python Packages. Honestly, my first reaction was “ugh, I need a book to understand packaging now?!” But, it’s a good book. Very clear text, well organized, and with specific recommendations instead of just surveying all of the options.

Dane’s book unblocked me and showed me not just which tools to use, but how to use them.

A plan comes together

I grabbed a copy to read, but Dane helpfully summarized the advice in the book in a post on Mastodon:

Dane Hillard:

it’s a short enough text :-) if you want an extreme TL;DR I’d say the following are the most immediate bang for the buck:

• Convert setup.py to setup.cfg or pyproject.toml
• Use pypa/build (unless you really like poetry or some such)
• Use tox to manage build/test/publish/etc.
• Hook up a job for cibuildwheel (if not pure Python)

I rely on three features of pbr heavily:

• using git to select the files to go into the source package, instead of listing them explicitly
• determining the version number for a package from the git tag, instead of having to update a source file inside the package
• producing a ChangeLog file automatically from the git history

I found setuptools-scm as an alternative for the first two requirements. It seems to produce slightly different pre-release versions than pbr, but that’s not as important for me as having a single source of truth for the version. It also seems to have the same effect of including all files in the git repository when building the source distribution.

There is no automated alternative for producing a change log, but there are other approaches and a slightly manual tool could be fine for something that sees so few changes as virtualenvwrapper. So, I put off worrying about that, for now.

Finally I was able to work out a plan.

1. Move as much of the metadata from the old configuration files into pyproject.toml as possible (spoiler alert).
2. Add setuptools-scm to the setup for the features I was losing by dropping pbr.
3. Switch from python setup.py sdist bdist_wheel to python -m build as a front-end for building packages, especially in my tox and CI job configurations.

The plan versus reality

I refuse to read standards documents (in this case PEPs) to figure out how to use a tool. So, back to the Python Packaging User Guide site, this time to find a description of that TOML file format.

The default for the tutorial uses hatchling instead of build for some reason. I found the “tab” showing the setuptools settings for the build-system section, though.

The description of the project section in the tutorial gave an example of the metadata and linked to the metadata specification (really more like a reference document), which was also helpful.

After a bit of studying, I came up with a patch that created pyproject.toml with most of the packaging settings that had been in setup.py, setup.cfg, and requirements.txt.

# pyproject.toml

[build-system]
 requires = ["setuptools", "setuptools_scm[toml]>=6.2"]
 build-backend = "setuptools.build_meta"

 [project]
 authors = [
   {name = "Doug Hellmann", email = "doug@doughellmann.com"},
   {name = "Jason Myers", email = "jason@mailthemyers.com"},
 ]

 classifiers = [
     "Development Status :: 5 - Production/Stable",
     "License :: OSI Approved :: MIT License",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Intended Audience :: Developers",
     "Environment :: Console",
 ]

 name = "virtualenvwrapper"
 description = ""
 dynamic = ["version"]
 keywords = ["virtualenv"]
 license = {text = "MIT"}
 readme = "README.txt"
 requires-python = ">=3.8"

 dependencies = [
     "virtualenv",
     "virtualenv-clone",
     "stevedore",
 ]

 # https://github.com/pypa/setuptools_scm/
 [tool.setuptools_scm]

 [project.urls]
 homepage = "https://virtualenvwrapper.readthedocs.io/"
 repository = "https://github.com/python-virtualenvwrapper/virtualenvwrapper"

 [project.entry-points."virtualenvwrapper.initialize"]
 user_scripts = "virtualenvwrapper.user_scripts:initialize"
 project = "virtualenvwrapper.project:initialize"

 [project.entry-points."virtualenvwrapper.initialize_source"]
 user_scripts = "virtualenvwrapper.user_scripts:initialize_source"

 [project.entry-points."virtualenvwrapper.pre_mkvirtualenv"]
 user_scripts = "virtualenvwrapper.user_scripts:pre_mkvirtualenv"

 [project.entry-points."virtualenvwrapper.post_mkvirtualenv_source"]
 user_scripts = "virtualenvwrapper.user_scripts:post_mkvirtualenv_source"

 [project.entry-points."virtualenvwrapper.pre_cpvirtualenv"]
 user_scripts = "virtualenvwrapper.user_scripts:pre_cpvirtualenv"

 [project.entry-points."virtualenvwrapper.post_cpvirtualenv_source"]
 user_scripts = "virtualenvwrapper.user_scripts:post_cpvirtualenv_source"

 [project.entry-points."virtualenvwrapper.pre_rmvirtualenv"]
 user_scripts = "virtualenvwrapper.user_scripts:pre_rmvirtualenv"

 [project.entry-points."virtualenvwrapper.post_rmvirtualenv"]
 user_scripts = "virtualenvwrapper.user_scripts:post_rmvirtualenv"

 [project.entry-points."virtualenvwrapper.project.pre_mkproject"]
 project = "virtualenvwrapper.project:pre_mkproject"

 [project.entry-points."virtualenvwrapper.project.post_mkproject_source"]
 project = "virtualenvwrapper.project:post_mkproject_source"

 [project.entry-points."virtualenvwrapper.pre_activate"]
 user_scripts = "virtualenvwrapper.user_scripts:pre_activate"

 [project.entry-points."virtualenvwrapper.post_activate_source"]
 project = "virtualenvwrapper.project:post_activate_source"
 user_scripts = "virtualenvwrapper.user_scripts:post_activate_source"

 [project.entry-points."virtualenvwrapper.pre_deactivate_source"]
 user_scripts = "virtualenvwrapper.user_scripts:pre_deactivate_source"

 [project.entry-points."virtualenvwrapper.post_deactivate_source"]
 user_scripts = "virtualenvwrapper.user_scripts:post_deactivate_source"

 [project.entry-points."virtualenvwrapper.get_env_details"]
 user_scripts = "virtualenvwrapper.user_scripts:get_env_details"

I said “most” of the settings, because I could move everything except for the executable scripts to be installed, for virtualenvwrapper the most important files in the package. Although the TOML input file can describe scripts, it expects them to be entry point-based console scripts written in Python, not anything else. The shell scripts in virtualenvwrapper are an edge case, obviously, and I don’t think the TOML layer should change to accommodate shell scripts, but the limitation meant I could not eliminate the setup.py file.

# setup.py

from setuptools import setup

setup(
    # Listing the scripts in pyproject.toml requires them to be python
    # entry points for console scripts, but they are shell scripts.
    scripts=[
        "virtualenvwrapper.sh",
        "virtualenvwrapper_lazy.sh",
    ],
)

Future work

For the change log, I took the output that had been produced by pbr and copied it into the Sphinx documentation set by hand, for now. I’ll investigate other tools to manage history like that in the future.

Some aspects of how the packaging changed broke the documentation build on readthedocs.org. I eventually fixed it, but that’s a story of its own, so it will wait for another blog post.

Final thoughts

I’m not entirely sure what triggered it, but apparently packaging Python projects has resurfaced as a topic of interest more broadly recently. In the course of the conversation about this on Mastodon, Juan shared a blog post from Pradyun Gedam from January of this year covering his Thoughts on the Python packaging ecosystem. I think I agree with a lot of what Pradyun says there.

I was around when Philip Eby first shipped setuptools and when it fell out of maintenance. I remember talking to Tarek Ziadé at a PyCon about creating distribute as a fork, and was happy when it was later folded it back into setuptools. I understand that Windows users were not well served by most of those tools, and how that helped the Conda ecosystem grow. And given that history, I can see why someone would be reluctant to choose a single path for implementation of packaging tools, even in a group called an “authority”. Still, the lack of clarity doesn’t help users. I would have liked to see guidance somewhere on packaging.python.org for which one tool to use and how to use it fully, without having to hunt around and assemble information from multiple sources.

• virtualenvwrapper
 
• python