My Version of Email Anywhere

posted in: Blog, The Technology Minutiae | 0

(Photo courtesy of Citrix Online user Robbie on the Great Wall of China via Flickr)

Why not just use gmail or some other existing provider? Existing providers are definitely the easy way to go. Unfortunately, I rarely go the easy way because there is usually something missing in the easy way–but if it works for you, go for it. Additionally, you may find new tools for accessing mail at gmail that may work well for you. When I first started fiddling with Linux, one of the first things I did was to play with email. At the time, email was the dominant application for the web (and it is still a fundamental internet application). I learned a key lesson from those fiddlings. Basically, I discovered that when my email was delivered to my mail box by my internet provider, the email no longer had some of the information that I wanted to use to further route and filter the email–the only way around those limitations was to run your own email server. For better or worse, we are all shaped by life’s lessons and I will tell you the implications of my lesson.

Soon, I was running my very own email server and I don’t see how I can go back. Now that we have a very mobile lifestyle, we use our virtual private server (VPS) Pluto to process email. For me, the advantages are that I can use any or all potential email addresses in my web domains–specifically, we can make up a new custom email address for every vendor we interact with, then if spam comes to that email address, we know who leaked it. I can filter, sort and route email however I want–I can even have email trigger custom programs on my server (the ultimate tool in faking a 24/7 presence). If an email disappears on my server, it was me that messed up and I believe I can fix the problem–if only Brenda will forgive me because for some strange reason it’s usually her email that disappears. I can decide how I access the email, who else can see my email, and I can decide how and where it is stored. The down side is that I have to figure out how to deal with security, spam and viruses, as well as how to make sure other email servers accept mail from our server. The latter is best accomplished by having a proper reverse DNS entry for the server which we do through our dyndns.com account.

Since my first fiddlings with email, I have used Postfix as my server’s MTA. When another server contacts our server with email to deliver, Postfix answers. Postfix checks all incoming connections and email to make sure they meet certain fundamental standards that all legitimate email servers will use–often spam email fails these checks. That was sufficient for several years, but then the spammers got better. Now, Postfix also checks the origination for each email against lists at several sites that compile lists of current spammers. Those lists are kept up almost to the minute by vigilant server admins who are constantly looking for spammers. Checking against those lists now blocks almost all spam. Next, the email is sent to Amavis for further scanning for spam and virus signatures. Amavis can use one or more subprograms to help do this. We have had very good luck blocking the latest viruses with ClamAV, but often use two different virus checkers in series just to be sure. The email that passes those checks is sent back to Postfix which then sends the email to the server’s local delivery agent, a program called Procmail.

Each user can set up Procmail (using their ~/.procmailrc file) to deliver their email as desired into multiple mail folders (we use the maildir format as opposed to the old mbox format). This is a good efficiency tool because you want the email delivered into the appropriate folder automatically. Brenda and I use completely different sorting approaches. Brenda uses folders that correspond to her web identities, I try to sort according to the fundamental nature of the email–direct communication, account updates, news info, etc. Both approaches have their strengths. Learning the syntax of .procmailrc is a little difficult, but some email programs provide a simple front-end that isolates the user from having to know the syntax. In addition, I have my .procmailrc send certain email message through Spambayes. Spambayes is trained when I designate certain email as spam and other email as good. It looks at specific signatures in each email and applies an algorithm based on a custom and evolving filter to determine the probability that a message is spam. Very, very little spam makes it past all those filters.

Now that the email is in personal mailboxes on the server, how do we read the email from anywhere? We employ a multi-tiered approach. First: we can access our email directly on Pluto, the VPS; second, we can synchronize our email with another more local server; and third, we can synchronize our email with a particular client on a personal computer. We would like each of those tiers to be as seamless as possible for the user. Here again, Brenda and I differ on our preferred email experience. Brenda likes a full-featured graphical interface with point and click, I prefer a svelte command-line-interface (CLI). The only difference between the two approaches is the top layer program. I use Mutt, a very powerful program with a very basic interface that requires user training to use effectively. For Brenda, I use a program called Squirrel Mail that serves up email through a user’s web browser–it’s a webmail program. Squirrel Mail has lots of plugins that one can install to add functionality–including a way to access the spam-detection controls of Amavis. Both ways of accessing email are always available for each user–I can choose to use Squirrel Mail and Brenda can use Mutt if she chooses.

For the first tier approach where we access email directly on the server, both Squirrel Mail and Mutt work. Squirrel Mail requires Pluto to run a web server, but we already have Apache running so we just need to configure Apache2 to serve up the Squirrel Mail site along with everything else it already does. Squirrel Mail require the server to run an IMAP program. I use Courier IMAP which runs in both a non-encrypted mode and a SSL encrypted mode. Squirrel Mail, which is on the server, can use the unencrypted IMAP. Then for privacy, we ensure that the user must access the Squirrel Mail through the Apache2 SSL server which encrypts everything. To use Mutt on the server (which accesses the email folders and doesn’t require IMAP), I first use SSH to establish a secure terminal on Pluto and then run Mutt in that terminal. If the computer I am using runs Linux, an SSH client is almost always available. In those rare instances where I am using a Windows machine, I can use Putty as an SSH client to establish a secure terminal. I have even used an SSH client on Brenda’s old Palm Treo to run Mutt on Pluto.

But what about when our internet connection is slow, flaky or non-existent. The magic software we use is called Offlineimap. This program allows us to sync our entire email directory tree on the remote server via the SSL IMAP to a local computer. Of course, one must have periodic internet access to make this work. But, that access can be very slow, since only changes since the last sync are transmitted. Offlineimap is very robust and I have never lost email due to a failure during Offlineimap syncs. Once the email is local, everything is available just like on the server, except that (as setup in .offlineimaprc configuration) I prepend all the folder names with .pluto so I know they are the ones that are synced to pluto. I can run Mutt on the local computer just like when accessing the server via SSH. Anything I read or move locally will show up as such on Pluto after the next run of Offlineimap. Thus both the local and server email status is identical and it doesn’t matter how or where I read or manipulate my email. To support Brenda locally, we run Courier IMAP on the local machine along with Apache and Squirrel Mail. She can then point her browser to the local machine to read her email.

There are some critical differences between the local and server setups that are important. We want all of our email to be sent from Pluto. Rather than setting up the client on the local machines with Pluto as the email server (which could cause problems when the internet connection is down), we use a feature of Courier IMAP on Pluto where we have a special mailbox called Outbox. Any email moved into that folder is found and sent by Pluto. Thus, on the local machine, we tell Mutt and Squirrel Mail to put copies of sent messages into the .pluto.Outbox folder. Then during the next Offlineimap sync, the email ends up in the Outbox on Pluto where Courier IMAP finds the email and sends it just as if we were writing the email on the remote server.

The third tier access to our email also works. I have a Nokia N810 Internet Ta that runs a Debian-based Linux called Maemo. Although a package for Offlineimap wasn’t available, it installed easily. Thus, I also have a complete mirror of my email on the N810. Mutt also runs on the N810 so I have my interface. We have also run other email programs that can sync a local version of email using IMAP–in many cases using the preferred secure SSL method.

So, in summary, we have the following functions and software on Pluto:

Receive email on server from senders: Postfix running on Linux OS
Filter spam: Postfix -> Amavis -> Clamav (and potentially other virus scanners) -> Amavis -> Postfix
Local delivery: Postfix -> Procmail
User sorting and filtering: Procmail -> Spambayes (Spambayes is optional) -> Procmail -> local email files in user home email directories

Support for reading email on Pluto:

Method 1: Courier IMAP  -> Squirrel Mail -> Apache SSL
Method 2: Courier IMAP over SSL
Method 3: SSH Server lets user login and then run Mutt to see files in home directory

Reading email on Pluto from remote computer:

Brenda: Web browser pointed at Squirrel Mail Web Site on Pluto
David: SSH Client then running Mutt on Pluto

Local machine functions and software for off-line reading:

Synchronizing email to local machine: Courier IMAP over SSL running on Pluto, Offlineimap on local machine with user .offlineimaprc configuration file

Off-line Reading of email:

Method 1: Courier IMAP -> Squirrel Mail -> Apache (Apache SSL if the connection is over a public network or wifi)
Method 2: Mutt (or SSH and Mutt if using another machine on the local network)

Sending email from Local machine:

Configure local Squirrel Mail and Mutt to use .pluto.Outbox for sent mail.
Configure Pluto Courier IMAP to use the special Outbox folder to send mail.

As you can see, this is a classic Linux service: lots of (not so) little programs strung together in a way to give the end user just the kind of experience they want. I could probably write an article on the details of how each step in the above summary is configured. Others probably don’t want to do exactly what we did, and they don’t have to. Over the years, I have spent a lot of time fine-tuning our email system, but for the most part, once configured everything just works. And it just works for years and years. This is a benefit of Open Source Software because if the program is good and the current maintainer gets bored and leaves to do something else, someone usually jumps in and keeps maintaining the software. Even I could keep the program running because I have the source code. Then, it’s just up to me to make small updates and changes to ensure our email system continues to support our mobile lifestyle.

Leave a Reply