API Critique¶
Recently I have been revisiting aspects of libESMTP and considering what may or may not be needed to bring the library up to date, since it is more than a decade since it has had any development. I felt it might be useful to lay out my observations and rationale for any changes. I will also consider creating language bindings since implementing these often gives some clarity on what might be missing from a C API.
I feel that libESMTP has remained free of any significant bitrot but there is definitely room for improvement.
API Design¶
I wanted libESMTP to have a small API. I reasoned that since sending an email should be a simple thing to do, the API should be small. Very early on I realised that an SMTP interaction with an MSA (Message Submission Agent) decomposed into one or more messages each with one or more recipients. This mapped very nicely into three objects representing each of these concepts and formed the basis of the API, each object being functionally distinct with their own methods to configure them. Furthermore it was very natural that the SMTP interaction, or session as I called it was a container for messages which in turn were containers for their recipients.
Another benefit that became clear was that this approach allowed the entire interaction to be controlled with a single API call.
At this point in the design the API consisted of functions to create and configure each of the objects. As an objective of libESMTP was to support a comprehensive set of SMTP extensions, it transpired that quite a few APIs were required to configure each of the libESMTP objects.
So far, so good. Originally I had intended that when a session was finished the application could iterate the messages and recipients within the SMTP session and discover whether messages had been transmitted successfully or whether any or off the of the recipients were rejected of failed for any reason. This proved to be satisfactory for simple command line type programs but not so good for interactive programs that might wish to present some sort of progress indication or report status as the session progressed. Therefore I introduced a callback mechanism to report events as the protocol progressed.
A further callback was added so that the SMTP interaction could be monitored while omitting the actual message transfer itself.
Another consideration was that although RFC 821 and RFC 822 and their successors are orthogonal to each other, that is message format is independent of the transmission protocol, there is slight crossover because an MTA is required to add a Received: header to messages in transit, and certain headers are required or prohibited in newly submitted messages. While libESMTP can handle most of this transparently to the application, it proved useful to add API calls to provide finer control of this.
The number of calls in the API was starting to grow. This had bothered me as it meant that the API would be harder to learn and went against the philosophy that sending a mail sould be a simple thing to do. Nowadays, this aspect bothers me less than it used to do, libESMTP still has a fairly small API by modern standards and each API has a specific purpose with few arguments.
ABI¶
One of my earliest decisions was that the ABI should be unaffected by internal changes in libESMTP, such as bug fixes or new features. The easiest way to achieve this was to make all libESMTP’s structures opaque so that their content was affected or accessed only through API calls. The actual structure definitions could be kept private to libESMTP’s code. This was easily achieved since C allows structure declarations without defining their content, as long as you only manipulate pointers to the structures. Since they are still treated as distinct types, errors in usage can be detected at compile time rather than by faulting at runtime.
It still surprises me two decades later that I still occasionally find libraries that fail to hide their structures as well as libESMTP, although there are cases where this is not possible.
Extensibility¶
The API has proven extensible since the associated objects can be modified, if necessary, without breaking the ABI. New methods are easily added by following the naming patterns for their respective API.
Ease of Use¶
I feel that libESMTP’s API is easy to use. Each object is logically distinct and directly relates to SMTP concepts and each method has a single well defined function acting on its object. Basic use cases require only a fraction of the full API to implement, in that respect I feel the original design goal is achieved.
As originally designed, the libESMTP API created objects and configured them as required, however few accessors were provided as it was expected that applications would only query status after the fact when a session completed. This has proven to be inconvenient in certain circumstances, for example, it is difficult to query a recipient object for its mailbox address. The lack of accessors also complicates the event callback interface. When writing a binding for languages whose objects support properties, the lack of accessors means properties are difficult to implement. This is discussed below for Python.
Naming Conventions¶
Namespace¶
Since C has no concept of namespaces for functions or variables, all the API
calls are prefixed with smtp_. I thought about using esmtp_ but that
seemed too long!
Objects and Methods¶
As with namespaces, C has no concept of objects or classes. structs being the
most similar construct available. libESMTP’s objects are implemented as structs
allocated and initialised by certain API calls, returning a pointer to the
application. Pointers for the three struct types were declared as
smtp_session_t, smtp_message_t and smtp_recipient_t (an additional type
was created for the ETRN extension).
On reflection, I feel I made a mistake in the API naming convention for the
object methods. I think I was trying to get them to read easily, since the
namespace_object_method convention tends to read backwards at times,
nevertheless this would probably have resulted in a more intuitive and more
easily memorised API.
These deficiencies make a simple mapping of API names to method names in object oriented language bindings particularly difficult and non-obvious. Had I designed the API nowadays, I would follow the more rigorous approach used in frameworks like GObject.
Internal Interfaces¶
Unfortunately, libESMTP did not hide its internal interfaces from public access, it is possible to link against any of its symbols at global scope. It is important to prevent linking against internal interfaces, since there may be unintended conflicts with symbols in third-party libraries or application code. The use of undocumented internal APIs can be a major barrier to development, especially if significant applications rely on them.
Using a consistent prefix for public symbols makes it easy, using wildcards, to create a linker map file to remove public references to internal calls, which do not use the namespace prefix. Without a consistent prefix, each publicly available symbol must be explicitly listed, this is error prone since it is easy to forget or misspell a name and there is no easy way to automatically check this.
Application Data¶
The application data API is problematic. The idea was to set an arbitrary pointer to data maintained by an application and associated with one of the libESMTP objects. I followed a design pattern which set a pointer and returned the old pointer which could then be deallocated or passed back to the application.
Unfortunately this approach requires an application to iterate all the libESMTP structures fetching back the pointer and deallocating it prior to destroying the libESMTP session. This significantly complicates use of libESMTP, is error prone and subject to memory leaks if not executed correctly. A better design would provide a release function to free or otherwise release the application data. libESMTP can guarantee to call this at the correct time, significantly easing the application design.
A well designed application data interface is important for many applications and is especially so for implementing language bindings, particluarly as neither side can control the other’s memory management strategy and since weak bindings may also be necessary.
Error Handling¶
Where API calls might fail they consistently return either zero or a NULL
pointer on failure, and non-zero/NULL values on success. The actual reason for
failure is set in a global (or thread local) variable. This was modelled after
the standard C library’s errno variable. I’m not sure whether this is a
satisfactory design but it does have precedent so it is a familiar pattern.
A second category of error occurs during a protocol exchange. This occurs when the peer reports a problem via a status code, for example, a recipient mailbox may not exist. libESMTP does not consider this an API failure, instead it provides APIs to allow the application to query status after the protocol session completes. As noted above the event callback may be used to report protocol status on-the-fly.
Type Safety¶
Although C is not a strongly typed language the libESMTP API is reasonably type-safe. There are, however, two parts of the public API and a few internal interfaces that use variadic functions, the event callback and the message header API. The latter is discussed further below. Unfortunately these interfaces cannot be validated for type-correct arguments either at compile time or runtime.
The internal interfaces may be verified by careful code review and static analysis, or by the compiler for the printf-like interfaces. Unfortunately this must also be done for every project using libESMTP. Since the variadic functions in the header API have only a few variations, a possible solution is simply to provide a variant for each function for each combination of types.
Callback Design¶
Ideally callbacks should be function closures, however C does not provide closures. This is resolved by providing an extra closure argument when the callback is registered and supplying that argument back to the callback. This is conventional in many C language APIs.
Event Callback Type Safety¶
Unfortunately the event callback’s function is somewhat overloaded. Similar events often have inconsistent arguments and semantics and the remaining events are special cases.
This design flaw is compounded since the varargs macros must be used to process the callback arguments which is error prone, and impossible to check either at compile time or run time. Errors introduced writing the event callback might only be detected when the application crashes.
This is particularly evident when trying to implement language bindings – translating the event callback into something appropriate to the host language becomes significantly more complex than it perhaps need to be, even for a closely related language like C++.
Fortunately callbacks other than the event callback do not suffer this issue.
Message Callback¶
I feel that I made my worst design decisions when implementing the message callback.
Memory management in the message callback is poor. The first argument to the callback is a pointer which can reference any arbitrary state the applicaton maintains while reading the message. However when libESMTP is finished reading the message if that pointer is not NULL it is passed to ‘free()’. This obliges the callback to use ‘malloc()’ but what if the application wants to reference its own private structures? Furthermore the application must use the malloc that pairs with the free libESMTP is linked against; sometimes this can be problematic and usually only discovered when the application faults.
Another problem is that the message callback design means the callback must allocate any state it requires on first call and so must check for first call on every call. Although the overhead of doing this is small, it complicates the code and makes it harder to read, doubly so since this behaviour was never properly documented. This is a pity because libESMTP’s internal logic knows both when it is about to use the callback for the first time and when it has finished reading the message.
Worse still, one of the arguments in the message callback is set to NULL to signal that the input stream should be rewound to the beginning of input; once more this strange behaviour is undocumented.
All of these objections could be improved by expanding the message callback into three open, read and close phases and eliminating the implicit free. The original API can be layered over this interface for backward compatibility.
File and String Reader Callbacks¶
libESMTP provides two standard callbacks for reading messages from a string or a FILE pointer. On reflection, I think the APIs to set these callbacks should have been provided as real functions in the library rather than as macros. Had they been functions, it would have been possible to remedy the objections discussed above in a new message API design and existing applications using an updated libESMTP would automatically use the new interface. The macros have made this impossible.
Iterators¶
It is a pity that libESMTP did not implement iterators in its API. In the libESMTP API, messages and recipients (and ETRN) may be iterated using the ‘enumerate’ APIs. These take a callback function to process each item.
For example, because C does not provide function closures, a callback function has no access to variables within the scope of the calling function. One way to work around this limitation is to put variables from the outer scope in a struct and pass a pointer to the struct as the callback argument. It works but it is always an ugly hack.
On reflection, a callback based design for iterating structures is really only a good approach in languages that provide function closures, particularly those that allow anonymous or lambda functions.
Iterators are almost always more convenient and intuitive to use and easier to use in language bindings.
Language Bindings¶
I think it is useful to consider providing language bindings for libESMTP. Doing so, it became obvious that certain aspects of libESMTP’s API design makes this more difficult than it needs to be.
Callbacks¶
The deficiencies in the event callback noted previously make binding to a language’s event mechanism more verbose than necessary and type safety is lost in the process; typically this would have to be implemented as a ‘big switch’ on the event type which selects the correct varargs statements to convert the arguments.
Host Language Objects¶
Because there is typically a one-to-one mapping between the bound language and libESMTP’s objects, they must be associated with each other. The application data APIs may be used to do this. The deficiencies in the application data API described above make this more difficult as the bound language’s objects may be reference counted or must be otherwise released to a garbage collector.
Names¶
Interfacing the libESMTP objects to the host language equivalents should be simple enough however, as previously noted, the lack of a rigorous naming convention means that the mapping from API name to the host language’s method or property names for each object may not always be clear.
Python¶
Python provides a comprehensive C API to integrate C libraries or code into applications either directly or via its module system. I feel that if all the issues described are addressed this would be much simpler to implement. In particular iterators and accessors, if available, would enable a much more Pythonic result.
Interfacing callbacks should not be problematic, even with libESMTP’s current API, however the glue code would face issues with correctly handling variadic callbacks.
For example the sequence
smtp_session_t session;
smtp_message_t message;
smtp_recipient_t recipient;
session = smtp_create_session ();
smtp_set_server (session, "example.org:587");
/* ... */
message = smtp_add_message (session);
smtp_set_reverse_path (message, "brian@example.org");
recipient = smtp_add_recipient (message, "douglas@h2g2.info");
/* ... */
smtp_start_session (session);
might be rendered in Python as
from libESMTP import Session, Message, Recipient
session = Session()
session.set_server ('example.org:587')
# ...
message = Message(session)
message.set_reverse_path("brian@example.org")
recipient = Recipient(message, 'douglas@h2g2.info')
# ...
session.start()
Unfortunately the constructors bear little relation to the C equivalents nor do the method API names have a clear mapping. Providing factory methods might be a good idea, for example
message = session.add_message()
which has the benefit of being closer to the C API, and therefore more intuitive, and requires fewer imports.
If libESMTP provided more accessors for some of its internal variables the reverse path mailbox for a message could be more naturally expressed as a property. Even without accessors, this would still be possible but would require the glue code to shadow the appropriate value which is both wasteful and error prone.
message.reverse_path = 'brian@example.org'
print(message.reverse_path)
The lack of iterators makes adding Python iterators or generator expressions difficult or impossible. For instance, it should be possible to check recipient status something like this:
session.start()
for message in session.messages():
print('From:', message.reverse_path)
for recipient in message.recipients():
print('To:', recipient.mailbox, recipient.status)
or even use a comprehension
status = {recipient.mailbox: recipient.status for recipient in message.recipients()}
Interfacing a Python file like object to a revised message callback should be a good test that the design is correct.
With these changes the example could be rewritten to be more Pythonic.
from libESMTP import Session
session = Session()
session.server = 'example.org:587'
# ...
message = session.add_message()
message.reverse_path = 'brian@example.org'
with open('message.txt','r') as filp:
message.fromfile(filp)
recipient = message.add_recipient('douglas@h2g2.info')
# ...
session.start()
for message in session.messages():
print('From:', message.reverse_path)
for recipient in message.recipients():
print('To:', recipient.mailbox, recipient.status)
C++¶
While the libESMTP API can be used directly in C++, a proper language binding would offer some benefits, especially since C++11 and later are arguably quite different to earlier versions of the language.
Following some of the design patterns in the STL would provide a much more natural API for modern C++, especially as libESMTP’s objects share some characteristics with STL containers.
Two STL patterns that come to mind are iterators and callbacks. libESMTP’s session and message objects share characteristics of std::forward_list an should follow their APIs. Allowing callbacks to use std::function values would permit use of lambda functions or any compatible object which overrides the ‘()’ operator.
Finally a C++ API could be created so that modern memory management practice is observed, such as construction in-place and the no naked new or delete rule.
Lua¶
Lua is an interesting case since it is a purely procedural language, however its data structure, the table, adds extraordinary flexibility in the choice of programming methodology. Lua also natively supports iterators.
Using Lua’s C API it would be almost trivial to add a one-to-one mapping of C APIs to Lua APIs but this would be a suboptimal approach. Lua tables can act both as an object and a type, via the use of metatables. Therefore Lua conventions are also better served by an object oriented approach but once again the difficulties outlined above become apparent.
GObject¶
GObject is a C framework rather than a language however it is a useful case to consider. When integrating libESMTP into a GTK+ program, having GObject APIs would be of some benefit. For example, making object state available as properties, mapping the libESMTP callbacks to GObject’s signal mechanism or taking advantage of GObject reference counting, introspection mechanisms and language bindings, such as Vala. libESMTP sessions could even be constructed using GBuilder XML files.
Other Languages¶
Resolving issues for C++, Python and Lua should simplify writing bindings for other languages, such as PHP, Ruby or D, especially if the basic set of bindings were provided as a standard part of the libESMTP release.
Conclusion¶
I feel it is useful to be self-critical of design decisions in various projects. On reflection I think I got libESMTP mostly right but there are deficiencies that are usefully remedied. This critique therefore serves as a loose plan for further development.
All the topics above can be resolved by extending the libESMTP API to create a strict superset and without having to alter the semantics of the existing API. Accessors for the libESMTP object state, where missing, are easily added and have no impact on the existing API. The event callback can be refactored into a set of type safe callbacks. An improved application data API has already been designed and implemented.
Where features are updated, the original APIs can be retained with deprecated status. This means the shared library can continue to work with existing applications but use of deprecated features in new programs can be flagged with compiler warnings or errors.
Although the API naming could be redesigned for better consistency and this could be done retaining the original names with deprecated status, I don’t think it is worth the effort. It would be confusing to have two names for many APIs and documentation, always a weak spot in many projects, would be overly complicated by this.
I feel that an important aspect of contemporary libraries is bindings for multiple languages. It should be straightforward, having addressed the issues outlined here, to provide official bindings for C++, Python, Lua and GObject. These should also serve as effective models for further bindings.
None of these are large or difficult changes to implement but should considerably enhance the API as libESMTP enters its third decade.
Brian Stafford, April 2021.