Handling High Email Volume with sup

Over the last year, the openstack-dev mailing list has averaged
2500 messages every month. Staying on top of that much email can be
challenging, especially with some of the consumer-grade email clients
available today. I’ve recently upgrade my email setup to use sup, a
terminal-based mail client that is helping me process the mailing
list, and even keep up with gerrit at the same time.

The Story Gets Worse Before It Gets Better

Last summer I moved all of my email processing from Google’s GMail
service to an account under my own domain hosted by FastMail. I liked
GMail’s web interface, but I had grown tired of some incompatibilities
with backup tools and other services. FastMail’s standard IMAP servers
fixed those issues, and I have been happy with the service.

This fall, however, after upgrading my laptop to Yosemite I started
seeing issues with mailing list threads being improperly combined so
that completely unrelated conversations all appear to be part of the
same thread. I traced the problem, thanks to several other Yosemite
users I know, to issues with Mail.app, the MUA that ships with
OS X. At first the problem was isolated to just a few threads. Then it
expanded to a lot of my mailing lists, and finally it started
affecting my inbox. There doesn’t seem to be any way to prevent Mail
from getting confused, or to fix it once it is.

I decided I needed a new mail client, and after reviewing the options
I decided I wasn’t going to find one that met all of my needs. The
problem first showed up with mailing lists, and while not all of the
poorly-threaded messages were from lists all of the threads involved did
include list messages. I tend to treat mailing lists differently that
my regular mail anyway, automatically filing it into folders using
mail filters and the batch reading it. So I decided to set up a second
email client just for reading mailing lists.

Looking At Options

One option I considered was going back to using GMail for the mailing
lists. I haven’t completely ruled this out, but I like having all of
my mail in one account so I don’t have to remember which one to use
when sending messages. I also don’t want messages sent directly to me
to end up in a never-never land by dropping them in an inbox I don’t
check frequently.

I used the FastMail web UI for a few months while I researched, and
it’s not terrible, but I felt I could be doing better. The key binding
support isn’t complete, so I do have to mix keyboard and mouse
commands. That left me looking for other desktop clients.

Several community members recommended mutt, but it looked more complex
than I needed. If I was going to move all of my email to a new
client, mutt would look like a better option. Since I am only reading
the mailing list this way, it felt like overkill.

A couple of people I talked to used sup, which is a terminal-based
app that has the same “archive and forget” behaviors of GMail. I like
that workflow, and the relative simplicity of the most basic setup, so
I spent some time this week configuring it to run on my laptop. After
a couple of false starts, mostly because I’m on OS X and the most
recent instructions are for Linux, I have it working and have been
using it for a few days. I’m already happier with the email workflow,
so I’m documenting the steps I went through to set it up in case
someone else wants to give it a try.

Set up offlineimap

The first step for most of the terminal-based clients is to install a
program to sync all of your email from the mail server to local
files. Desktop clients like Mail.app include this feature as a
built-in, but most terminal apps don’t. Offlineimap is a good
solution for syncing both directions, so that messages you read
locally are marked as read on the server as well.

Homebrew includes a package, so the first step is to install the code
through brew:

$ brew update
$ brew install offlineimap

The next step is to set up the configuration file. The documentation
on the project site gives a complete description of all of the
options, including setting up accounts and repositories. I’ll just
focus on a few of the unusual values I changed.

I already have the mail server configured to sort messages into
mailboxes based on the mailing list id, all under an OpenStack
parent folder. I also have some OpenStack mail archive folders there
for messages that didn’t go through a mailing list. I only want those
folders synced, so I set folderfilter to include those and my sent
mail box.

folderfilter = lambda foldername: foldername.startswith('INBOX.OpenStack') or foldername in ['INBOX.Sent Items']

I had trouble adding the resulting maildirs to sup because of the
spaces in the names, so I also added a name translation function to
nametrans to remove the spaces. While I was at it, I removed the
INBOX. prefix.

nametrans = lambda foldername: re.sub('^INBOX.', '', foldername).replace(' ', '_')

The result is a copy of each of my mailboxes saved under ~/Mail in
maildir format. Using maildir instead of mbox means both sup and
offlineimap can modify different messages in the same mailbox without
conflict. It’s also faster to sync the messages than mbox is.

The next step was to run offlineimap to have it download all of my
current messages. This took a while, since I have about 6 months of
list traffic archived (another nice feature of FastMail is the
auto-purge setting on each mailbox, which means I don’t have to delete
old list messages myself).

Doing the initial sync manually will help uncover connection issues or
other configuration problems, but you don’t want to do it by hand
every time you run it. Instead, you want to set up a cron or launchctl
job to sync your mail regularly. The Homebrew package for offlineimap
includes instructions for setting up a launchctl job, so that’s what
I did.

$ ln -sfv /usr/local/opt/offline-imap/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.offline-imap.plist

Set up msmtp

The IMAP protocol is useful for reading messages and accessing mail
boxes on a server, but to send messages you need an SMTP client. The
sup wiki explains how to configure msmtp, so that’s what I went with.

$ brew install msmtp

FastMail uses authentication for its SMTP servers, so only customers
can use them to send mail directly. msmtp was happy to access my
keychain for credentials, so I didn’t have to include my credentials
in clear-text in that file. The only tricky part of setting that up is
the msmtp docs were not quite right about how to create the keychain
entry. The account entry in the keychain should match the user
entry from the msmtp configuration file.

After configuring, I used this command to test:

$ echo test | Mail -s "test on $(date)” my-address@here

I did find that the mail server would deliver some of the messages to
individual addresses, but not to the mailing list. I’m not sure if the
list’s spam filter was blocking them, or if it was happening at
FastMail. I found that setting maildomain to my domain fixed the
problem.

maildomain     doughellmann.com

Set up sup

sup is written in Ruby, so I had to install Ruby and a few other
related tools before it would work. I’m not that familiar with any of
the tools, so I followed instructions I found in the sup
documentation for installing on OS X. It’s possible there is a
simpler way to do this part.

First, install a recent version of ncurses with wide character
support:

$ brew tap homebrew/dupes
$ brew install ncurses
$ brew doctor
$ brew link ncurses --force

Next, following https://rvm.io/rvm/install, install rvm and use it
to install ruby:

$ curl -sSL https://get.rvm.io | bash -s stable --ruby
$ rvm install 2.2.0
$ rvm use 2.2.0

Now it’s possible to install the gems needed by sup. Installing them
all together failed, but installing a couple of the dependencies first
and then installing sup worked fine.

$ gem install xapian-ruby
$ gem install gpgme
$ gem install sup

At this point all of the software is installed, so the next step is to
configure sup to tell it where your email is and how to work with
it. The sup-config command offers a step-by-step setup guide. I
used that for most of the settings, but did not add the mail sources
(more on why in a bit).

$ sup-config

After the basic configuration was done, I edited
~/.sup/config.yaml so that sup would save my outgoing mail to the
Sent_Items mailbox managed by offlineimap and so it would update the
flags on messages (to indicate they had been read, for example) in the
source maildirs. I also updated the sendmail command for the account
to point to msmtp.

:sent_source: maildir:/Users/dhellmann/Mail/Sent_Items
:sync_back_to_maildir: true
:accounts:
  :default:
    :name: Doug Hellmann
    :email: doug@doughellmann.com
    :sendmail: "msmtp -t --read-envelope-from"
    :signature: "/Users/dhellmann/.signature"
    :gpgkey: ''

Next, I used sup-add to add the mail being downloaded by
offlineimap. First I created the source for the sent folder, using
a special option to tell sup not to add messages from that folder to
my inbox:

$ sup-add --archive maildir:/Users/dhellmann/Mail/Sent_Items

Then I added the other mailboxes without the –archive flag,
giving some of them a flag to automatically add labels to the messages
in that mailbox. For example, the mailbox with all of the messages
from gerrit was added with:

$ sup-add -l code-review maildir:/Users/dhellmann/Mail/OpenStack.Code_Review

After all of the mailboxes were added, I ran sup-sync-back-maildir
to reassure sup that it should sync the status of messages back to the
maildir copy, so offlineimap could then update their status on the
mail server.

$ sup-sync-back-maildir -u

And finally I imported all of the messages into sup’s index using
sup-sync. Because all of the messages were read and archived
already, I used the –read and –archive options to set them
that way in the index as well (otherwise I would have had something
like 25,000 unread threads in my inbox).

$ sup-sync --read --archive

Now running sup gives a view of the unprocessed messages. I can
navigate through them using vi keystrokes, kill uninteresting threads,
archive read messages, and reply as needed. See the sup home page for
screenshots and documentation about navigation commands.

Results

Even though all of the messages are separated into different folders
on the mail server, sup shows them all as part of one inbox. This lets
me pay attention to bug report emails and gerrit notifications again,
which I had been ignoring because of the volume. Since they’re
trickling in a few at a time, and interleaved between other messages,
I’m finding it easier to keep up with them. If that changes, I may
drop those folders from my offlineimap configuration and go back to
looking for updates online separately.