Retrospective

The following is an account of my experience with libESMTP as it approaches about 20 years since I started work on it.

During the late 1990s and early 2000s I was working on email based technologies and had consequently taken an active interest in a number of IETF working groups, including those revising RFC 821 and RFC 822 which were at the time about 16 years old. I was also fortunate to have had a supportive employer at that time who was able to introduce me to one of the UK based IETF participants and who allowed me to attend a number of IETF meetings internationally. My first IETF was, appropriately enough for a fan of Douglas Adams’ books, IETF 42 in Chicago.

IETF participation was, in retrospect, an accelerated learning and profoundly useful experience. I was able to meet and discuss protocol development with some of the key personalities from the early days of internet development, both on a technical and philosophical level. By becoming involved in standards development I also learned how to critically read and, more importantly, understand RFCs and standards in general.

Having acquired a good working knowledge of, and the rationale for, various IETF protocols I wanted to find a way to encapsulate that knowledge in a useful way. Consequently I started work on libESMTP as I was unhappy with coding quality or performance of the programs and libraries which were available at that time.

Efficiency

A significant design goal was to be efficient on the network; most programs available at that time performed very poorly if network latency was high, for example on the wide-area network, or if the line turn-around time was high as is typical of dial-up half-duplex modems, leading to long delays as the TCP handshake took place.

Consequently libESMTP has always, by design, supported buffering and pipelining SMTP commands which maximises network packet size, reduces the number of TCP handshakes, and with pipelining packs multiple SMTP commands into a single network packet. This made dramatic improvements to submission times on dial up connections, from minutes to seconds in some cases.

A 14k4 half-duplex modem takes about 100-300ms to turn round the line. Similar figures apply to the 56k modems also available back then. A TCP handshake takes two line turnarounds followed by one more to transmit the next packet. I had made a few measurements which showed that for a typical SMTP client there were small bursts of data, usually no more than 20-70 bytes followed by long idle periods - sometimes as much as one second. Overall data transfer was poor at a few hundred bytes per second, despite the modem’s underlying bit rate. On the same connection libESMTP was able to fully saturate the bandwidth with data, performing line turnaround only as strictly necessary.

Code Quality

I put considerable effort into compiling strictly ISO-C / Posix / SUS compliant code, with maximum pedantic compiler errors enabled, and a zero tolerance approach to any compiler warnings. It was painful adjusting to this at first but it paid off and I would recommend anyone to do the same and see it through.

Release

I made the first release of libESMTP in February 2001.

I had initially planned to use Sourceforge to host the project, however libESMTP used OpenSSL and the Crypto Wars were well under way at the time. Unfortunately Sourceforge was hosted in the US and I wanted to avoid issues surrounding US crypto law, so I looked round for another host and found UK Linux. Better still, UK Linux had free hosting for open-source projects.

Reaction

I feel that libESMTP has always been a bit niche. Most users seem to have just quietly got on with it with nothing much to say, positive or negative. An early adopter was the Balsa email client, in part because I submitted the initial patches and in part because it meant that, at least for a while, it had better email submission than Netscape Communicator. A good number of early contributions came from the Balsa project, especially in the TLS support.

A significant early contribution came from (if memory serves) Cambridge University in the UK where libESMTP has been put through a static analysis program in development there. It had found only two issues, both potential buffer overflows. I’m told this was the best result from the open-source projects they had analysed at the time! That was quite a boost for my ego too. More recently, the Clang static analyser gave it a clean bill of health.

Other than that, there was a small trickle of patches from other sources for bugs that came to light but after a while libESMTP became quite stable and the trickle of patches dried up completely.

Because libESMTP supported NTLM authentication with MS Exchange servers I unexpectedly gained a reputation for being an expert in the area. If I did it was flattering but undeserved, all I did was search online until I found an account of how to do NTLM and coded up something to match. To my surprise, it worked. It was mostly an issue of bit packing and endianness, Microsoft’s brand of Unicode and a weak MD4 based cipher. Then again, NTLM caused people in the open source world problems back then so maybe I was the expert.

I suspect my overall experience is typical for open source projects; people tend to be more vocal when things do not work as expected.

What I was not prepared for was the abuse.

Ad-Hominem

A lot of mail I received was not bug-fixes, critique, or discussion one might expect but rude or abusive. This was a surprise and deeply upsetting. Bear in mind that I have never been paid for any of the work I put into libESMTP, except when someone once sent me a £10 donation via PayPal less what ever their commission was at the time.

I had put a lot of time into development and testing into libESMTP and it worked fairly well and it was free. I thought people might be grateful - many were - but it certainly did not feel like that in 2001. Nobody wants to read through ad-hominem attacks to find the useful stuff, or that there was none. I probably missed out on a lot of helpful advice because of that. I moved on to other things and put less effort into libESMTP than I might otherwise have done.

If anyone has ever detected a somewhat, errmm…, stroppy tone in the bug reporting section, that’s due to a certain level of exasperation. I received many, often rude, messages about things which were explained on the old web-site or where users’ expectations were flatly contradicted by the RFCs. After I updated the bug reporting page, those messages stopped coming.

When I think back from the on-line experience in 2020, all of this is much less surprising and nowadays perhaps I’d just let it wash over me. The abuse was a small sample of what was to come on the yet-to-evolve social media phenomenon. Facebook, Twitter and other social media sites are as much vehicles of trolling and misinformation as they are of rational thought and discussion, or more so.

Since many projects have added a code-of-conduct to their web-sites I suspect my experience is not unique.

Hosting

Originally there was a libESMTP mailing list, but unfortunately this was before the days of effective spam filtering and the mailing list quickly became overwhelmed. I literally couldn’t clean the junk mail fast enough. That fell by the wayside pretty early on. With hindsight I should probably have looked around for a host providing a subscription based mailing list. So I used my personal email address for libESMTP issues but this was not particularly satisfactory.

The libESMTP website continued to work well, however, and libESMTP had pretty modest hosting requirements - basically just the web pages and a few tarballs. Nevertheless, libESMTP has always suffered by not having a proper issue tracking system.

Documentation

I believe the libESMTP API is fairly logical and straightforward to use, but any project relies on good documentation. Originally I had the idea that the API Reference should be available in different formats so I chose to write it using Docbook XML markup. Without a purpose designed tool, this turned out to be a world of hurt and pain. There were, however, XSL-T stylesheets available to translate the output to HTML and XSL-FO. I was never able to make use of the latter but, with a little customised XSL-T and CSS, the HTML output looked good.

Because Docbook is difficult to write with a programmer’s editor, the API document progressed up to a point and stalled somewhat. Although the majority of public APIs were documented, quite a few remain to this day to be done and none of the auth-client API was ever documented. The example program provided with libESMTP or reviewing the source was all there was for a good portion of the API. I can’t say for sure if this was ever a show-stopping problem but I really have no idea either way.

Subsequent Development

The bulk of development on libESMTP was from about 2001 to 2006 with things settling down after that; I was receiving few or no bug reports and the last major release was about 2010. I feel this is because libESMTP has been pretty solid and reliable.

After about 2004, having changed employment I did not have the freedom to pursue IETF and opensource activity I previously had enjoyed. As development work became more demanding and had shifted away from mail based technologies, time and motivation to further develop libESMTP were in short supply.

Website

Sometime in the early 2010s my libESMTP email address stopped working, it was mostly junk mail so I barely noticed. After a few more years the website disappeared too. To be fair I was probably notified but the registered email address was long since obsolete.

I first found out about the disappearance at a job interview! I had mentioned libESMTP on my CV and as part of the prep work my interviewer had looked it up and it was gone! Oops! Fortunately libESMTP had been well established as part of most Linux distributions by that time and it was incorporated by an open source project organisation was already using. But it was an awkward moment. To be fair I had had almost 15 years of free ISP services, so I’m not complaining. I should have moved to GitHub in its early days. I got the job.

Recent Work

Some years ago I set up the GitHub account to host libESMTP, but the lack of spare time and experience with Git (I had been using CVS and SVN previously) meant that I more or less forgot about it, even when Microsoft subsequently acquired GitHub. However since then I had been using git and GitLab professionally so I decided to use these for personal projects too.

GitLab being a heavyweight for personal use I adopted Gogs instead. Currently I run Gogs on a Raspberry Pi 2 with a RAID-10 array of USB sticks, which sounds a bit crazy! Sometimes it’s a bit sluggish because the RPi’s USB and ethernet support is actually a 4-Port USB hub with a built-in ethernet dongle, but this combination works extremely well and I don’t have to worry about energy usage.

Modernisation

Late last year (2019) I decided to modernise the libESMTP distribution. Some time ago during a major move, I had found an old hard drive with all the libESMTP tarballs back to version 0.8. I was able to recreate a git repository from the tarballs and used that as a starting point. I probably still have the old SVN repository but it isn’t really worth the effort to migrate. I’ve been slowly updating things since then.

SASL

Back in 2010 I added support for GNU SASL as an alternative to libESMTP’s native SASL implementation and released this as version 1.0.7rc1. I never had any feedback on this and I have no idea if anyone is using it. It’s currently in git on the gsasl branch.

Autotools

A constant thorn in my side has been GNU autotools. I hate autotools. I never liked them, in part because of m4 quoting and in part because the generated configure script was bigger than the code it was building, causing significant bloat in the distribution. And all those commands that must be run in just the right order with just the right options. After a few updates to some of the associated files I was finding it difficult to change things; I felt the infrastructure was rather fragile, especially for out-of-tree builds or cross-builds if things weren’t set up just-so. Also many projects were migrating to Meson and I was losing the will to live. Enough was enough.

The first thing to do was migrate to Meson/Ninja.

Obsolete Code

When libESMTP was originally written, getaddrinfo() had only recently been introduced and was not widely supported. Rather than polluting the source with lots of #ifdefs I wrote a simple IPv4-only emulation round gethostbyname() and a few other network functions. These and a few other items are unnecessary in contemporary systems and are removed. Other functions that supplemented a local C library are removed or consolidated and there are a few changes to ensure SUS/Posix compliance for things like header files and so on.

Compliance

There are a few updates to RFCs in the intervening years. libESMTP needs review of the code to update references to current documents and to ensure code complies with anything that has changed. Because of how interoperability and changes are managed in RFC development this is likely to be a simple task.

Unit Tests

I have added a branch for unit tests on some of the libESMTP internals. I was bitten by a major regression sometime around 2008 causing X.509 certificate validation to fail. The fix was simple enough but this sort of thing should not happen.

Documentation

Most of the recent work has been to gradually bring documentation up to modern conventions. For example, facilities like Github pages or Read the Docs.

I decided to move the API documentation into comments in the source, similar to gtkdoc conventions. This proved to be much harder to do than I expected. Gtk-doc is a pretty flaky toolchain in my experience. It’s fine for GObject based programs but falls apart quickly when presented with something similar to GObject code but not quite - like libESMTP.

HotDoc was much better but I still wasn’t happy with some aspects of the result. But HotDoc is promising, especially as a gtkdoc replacement, and it stays firmly on my radar.

Sphinx gives the results I was looking for, especially with the Read the Docs theme. The only problem is extracting the documentation comments from the source. I settled on using the kernel-doc program from the Linux Kernel distribution since it does most of what I want. The bits I don’t like I can live with for now.

Migrating to Sphinx means a preference for ReStructuredText over Markdown. Personally, I have a mild preference for Markdown as I find it easier to write and the end result looks more like a plain text file than the former. For the documentation comments in the source code this makes little difference but for other files like this one I just find Markdown easier.

In Conclusion

I am hopeful libESMTP will continue to be useful in the future. libESMTP now has a modern build system and the project can make better use of GitHub’s facilities.

I am extremely proud of libESMTP. Even after 20 years I believe it is unparalleled as an SMTP client. Although it is in need of some modern authentication mechanisms, the core library is robust and, I believe, despite its age is still complies with modern standards.

And if you’re ever stuck on the end of a limited bandwidth network or dial up is your only option, libESMTP is the only show in town! Well, I would say that, wouldn’t I?

Brian Stafford, June 2020.