Handling High Email Volume with sup

Originally posted Mar 17, 2015 · 9 min read

Over the last year, the openstack-dev mailing list has averaged 2500 messages every month. Staying on top of that much email can be challenging, especially with some of the consumer-grade email clients available today. I’ve recently upgrade my email setup to use sup, a terminal-based mail client that is helping me process the mailing list, and even keep up with gerrit at the same time.

The Story Gets Worse Before It Gets Better

Last summer I moved all of my email processing from Google’s GMail service to an account under my own domain hosted by FastMail. I liked GMail’s web interface, but I had grown tired of some incompatibilities with backup tools and other services. FastMail’s standard IMAP servers fixed those issues, and I have been happy with the service.

This fall, however, after upgrading my laptop to Yosemite I started seeing issues with mailing list threads being improperly combined so that completely unrelated conversations all appear to be part of the same thread. I traced the problem, thanks to several other Yosemite users I know, to issues with Mail.app, the MUA that ships with OS X. At first the problem was isolated to just a few threads. Then it expanded to a lot of my mailing lists, and finally it started affecting my inbox. There doesn’t seem to be any way to prevent Mail from getting confused, or to fix it once it is.

I decided I needed a new mail client, and after reviewing the options I decided I wasn’t going to find one that met all of my needs. The problem first showed up with mailing lists, and while not all of the poorly-threaded messages were from lists all of the threads involved did include list messages. I tend to treat mailing lists differently that my regular mail anyway, automatically filing it into folders using mail filters and the batch reading it. So I decided to set up a second email client just for reading mailing lists.

Looking At Options

One option I considered was going back to using GMail for the mailing lists. I haven’t completely ruled this out, but I like having all of my mail in one account so I don’t have to remember which one to use when sending messages. I also don’t want messages sent directly to me to end up in a never-never land by dropping them in an inbox I don’t check frequently.

I used the FastMail web UI for a few months while I researched, and it’s not terrible, but I felt I could be doing better. The key binding support isn’t complete, so I do have to mix keyboard and mouse commands. That left me looking for other desktop clients.

Several community members recommended mutt, but it looked more complex than I needed. If I was going to move all of my email to a new client, mutt would look like a better option. Since I am only reading the mailing list this way, it felt like overkill.

A couple of people I talked to used sup, which is a terminal-based app that has the same “archive and forget” behaviors of GMail. I like that workflow, and the relative simplicity of the most basic setup, so I spent some time this week configuring it to run on my laptop. After a couple of false starts, mostly because I’m on OS X and the most recent instructions are for Linux, I have it working and have been using it for a few days. I’m already happier with the email workflow, so I’m documenting the steps I went through to set it up in case someone else wants to give it a try.

Set up offlineimap

The first step for most of the terminal-based clients is to install a program to sync all of your email from the mail server to local files. Desktop clients like Mail.app include this feature as a built-in, but most terminal apps don’t. Offlineimap is a good solution for syncing both directions, so that messages you read locally are marked as read on the server as well.

Homebrew includes a package, so the first step is to install the code through brew:

$ brew update
$ brew install offlineimap

The next step is to set up the configuration file. The documentation on the project site gives a complete description of all of the options, including setting up accounts and repositories. I’ll just focus on a few of the unusual values I changed.

I already have the mail server configured to sort messages into mailboxes based on the mailing list id, all under an OpenStack parent folder. I also have some OpenStack mail archive folders there for messages that didn’t go through a mailing list. I only want those folders synced, so I set folderfilter to include those and my sent mail box.

folderfilter = lambda foldername: foldername.startswith('INBOX.OpenStack') or foldername in ['INBOX.Sent Items']

I had trouble adding the resulting maildirs to sup because of the spaces in the names, so I also added a name translation function to nametrans to remove the spaces. While I was at it, I removed the INBOX. prefix.

nametrans = lambda foldername: re.sub('^INBOX.', '', foldername).replace(' ', '_')

The result is a copy of each of my mailboxes saved under ~/Mail in maildir format. Using maildir instead of mbox means both sup and offlineimap can modify different messages in the same mailbox without conflict. It’s also faster to sync the messages than mbox is.

The next step was to run offlineimap to have it download all of my current messages. This took a while, since I have about 6 months of list traffic archived (another nice feature of FastMail is the auto-purge setting on each mailbox, which means I don’t have to delete old list messages myself).

Doing the initial sync manually will help uncover connection issues or other configuration problems, but you don’t want to do it by hand every time you run it. Instead, you want to set up a cron or launchctl job to sync your mail regularly. The Homebrew package for offlineimap includes instructions for setting up a launchctl job, so that’s what I did.

$ ln -sfv /usr/local/opt/offline-imap/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.offline-imap.plist

Set up msmtp

The IMAP protocol is useful for reading messages and accessing mail boxes on a server, but to send messages you need an SMTP client. The sup wiki explains how to configure msmtp, so that’s what I went with.

$ brew install msmtp

FastMail uses authentication for its SMTP servers, so only customers can use them to send mail directly. msmtp was happy to access my keychain for credentials, so I didn’t have to include my credentials in clear-text in that file. The only tricky part of setting that up is the msmtp docs were not quite right about how to create the keychain entry. The account entry in the keychain should match the user entry from the msmtp configuration file.

After configuring, I used this command to test:

$ echo test | Mail -s "test on $(date)” my-address@here

I did find that the mail server would deliver some of the messages to individual addresses, but not to the mailing list. I’m not sure if the list’s spam filter was blocking them, or if it was happening at FastMail. I found that setting maildomain to my domain fixed the problem.

maildomain     doughellmann.com

Set up sup

sup is written in Ruby, so I had to install Ruby and a few other related tools before it would work. I’m not that familiar with any of the tools, so I followed instructions I found in the sup documentation for installing on OS X. It’s possible there is a simpler way to do this part.

First, install a recent version of ncurses with wide character support:

$ brew tap homebrew/dupes
$ brew install ncurses
$ brew doctor
$ brew link ncurses --force

Next, following https://rvm.io/rvm/install, install rvm and use it to install ruby:

$ curl -sSL https://get.rvm.io | bash -s stable --ruby
$ rvm install 2.2.0
$ rvm use 2.2.0

Now it’s possible to install the gems needed by sup. Installing them all together failed, but installing a couple of the dependencies first and then installing sup worked fine.

$ gem install xapian-ruby
$ gem install gpgme
$ gem install sup

At this point all of the software is installed, so the next step is to configure sup to tell it where your email is and how to work with it. The sup-config command offers a step-by-step setup guide. I used that for most of the settings, but did not add the mail sources (more on why in a bit).

$ sup-config

After the basic configuration was done, I edited ~/.sup/config.yaml so that sup would save my outgoing mail to the Sent_Items mailbox managed by offlineimap and so it would update the flags on messages (to indicate they had been read, for example) in the source maildirs. I also updated the sendmail command for the account to point to msmtp.

:sent_source: maildir:/Users/dhellmann/Mail/Sent_Items
:sync_back_to_maildir: true
:accounts:
  :default:
    :name: Doug Hellmann
    :email: doug@doughellmann.com
    :sendmail: "msmtp -t --read-envelope-from"
    :signature: "/Users/dhellmann/.signature"
    :gpgkey: ''

Next, I used sup-add to add the mail being downloaded by offlineimap. First I created the source for the sent folder, using a special option to tell sup not to add messages from that folder to my inbox:

$ sup-add --archive maildir:/Users/dhellmann/Mail/Sent_Items

Then I added the other mailboxes without the --archive flag, giving some of them a flag to automatically add labels to the messages in that mailbox. For example, the mailbox with all of the messages from gerrit was added with:

$ sup-add -l code-review maildir:/Users/dhellmann/Mail/OpenStack.Code_Review

After all of the mailboxes were added, I ran sup-sync-back-maildir to reassure sup that it should sync the status of messages back to the maildir copy, so offlineimap could then update their status on the mail server.

$ sup-sync-back-maildir -u

And finally I imported all of the messages into sup’s index using sup-sync. Because all of the messages were read and archived already, I used the --read and --archive options to set them that way in the index as well (otherwise I would have had something like 25,000 unread threads in my inbox).

$ sup-sync --read --archive

Now running sup gives a view of the unprocessed messages. I can navigate through them using vi keystrokes, kill uninteresting threads, archive read messages, and reply as needed. See the sup home page for screenshots and documentation about navigation commands.

Results

Even though all of the messages are separated into different folders on the mail server, sup shows them all as part of one inbox. This lets me pay attention to bug report emails and gerrit notifications again, which I had been ignoring because of the volume. Since they’re trickling in a few at a time, and interleaved between other messages, I’m finding it easier to keep up with them. If that changes, I may drop those folders from my offlineimap configuration and go back to looking for updates online separately.