Some might dread the feature of “read notifications” supported by certain MUAs; some call it an “invasion of privacy”; and yet, it can also be useful in certain situations:

When a message is read or seen in a MUA supporting this extension, the programme emits a notification back to the sender saying something along the lines of “your message … was read on …”. This is good to know, especially in times when you cannot wait for the failed-delivery-notification that follows four to five days of unsuccessful (but furious) attempts of some delivery agent, assuming it doesn’t get trashed as spam.

Such a read notification is logically a reply to the original message, isn’t it?

The RFC 680 proposed in April 1975 defines the header References as a way to point to “other correspondence which this message references”. This header, along with In-Reply-To (defined in the same RFC), is commonly used in every-day mail traffic to refer to previously exchanged messages, and enables mail readers to thread separate messages together into coherent conversations (it takes a human to remove the coherence, the technical aspect is infallible).

Cut.

Microsoft was also founded in April 1975, and it took them 20 years to barely manage to squeeze through the Internet door without the proverbial foot in it. They published a browser and several e-mail programmes, and it always appeared as if they tiredlessly tried to be different from the rest, attempting to form a clique of users, a Microsoft league in which to increase their revenue through network effects. Sounds bad, is bad, but yet again, they managed, through unimaginable feats of entrepreneurial genius and ruthless behaviour.

Cut.

In 1982, STD11 declared the aforementioned In-Reply-To and References headers as standards. At that time, Microsoft software didn’t even know what a computer network was.

Cut.

Does it come as a surprise that read notifications sent by Microsoft e-mail programmes, such as Microsoft Outlook do not make use of either of these standard headers to tag read notifications they send?

Instead, Microsoft pushes Thread-Topic and Thread-Index, which are undocumented and thus probably only work in a Microsoft-only context.

How am I supposed to assume anything else than Microsoft actively trying to oppose standards.

Anyone who boycotts standards is hindering progress and should be left behind. It’s good to see that the Internet society seems to follow that trend more and more.

Update: I found a way to extract the data to recreate the In-Reply-To with procmail. I don’t see a way to do the same for the References header. Also, I’ve only verified that this works for message disposition notifications from Outlook 2003, although I expect it to work for other, similarly crippled MUAs too.

If you deal with SMTP servers, you probably know swaks, the “Swiss Army Knife for SMTP”. Great tool for anything related to sending mails.

Today, I found its counterpart for receiving mails:

python -m smtpd -n -c DebuggingServer localhost:1025

that binds a no-frills SMTP server to the specified socket which does nothing but talk SMTP to connecting clients and print “received” messages to stdout, e.g.:

---------- MESSAGE FOLLOWS ----------
Date: Fri, 22 May 2009 21:21:40 +0200
To: madduck@madduck.net
From: madduck@lotus.madduck.net
Subject: test Fri, 22 May 2009 21:21:40 +0200
X-Mailer: swaks v20061116.0 jetmore.org/john/code/#swaks
X-Peer: 127.0.0.1

This is a test mailing

------------ END MESSAGE ------------

Sweet, very sweet!

NP: Porcupine Tree: Signify

While drafting emails to be sent out as part of my research, I just had an idea about a new mail header field, let’s call it X-Respond-By. This header field allows the sender to specify when s/he requests/requires/needs a response. The recipient could make use of this in multiple ways:

• automatic expiry of messages whose deadlines have passed, possibly with an acknowledgement issued to the sender,
• delayed delivery of messages until they are due,
• prioritisation of message display within the user agent to highlight messages which require action.

Obviously, care would have to be taken to prevent abuse, e.g. by some senders who make all their messages require responses on the next day. One way to achieve this would be to give the recipient sender-specific policies to ignore the field, extend it by a fixed period, or simply reject messages with the field set.

I already use a tickler system (which works locally as well as remotely) inspired by the GTD concept of a tickler file, which allows me to have email messages delayed to a given point in the future. I could very well imagine how such an approach could extend into a cooperative communication method.

I am looking forward to your comments.

Update: Jan Hudec wrote in to tell me about the Reply-By field used by Outlook, which sends messages with those headers:

X-Message-Flag: Follow up
Reply-By: Wed, 10 Sep 2008 12:00:00 +0200

According to Jan, this will

will mark the message with specific symbol,and at the time specified it will pop up a notification in the same way as it does for calendar events and after that it will show the mail in red in the index until it’s marked “completed”.

He also notes that Reply-By stems from X.400 via RFC 2156 and has indeed been standardised by RFC 4021.

I shall investigate this further when I have time.

NP: Foo Fighters: Echos, Silence, Patience & Grace

It happened a number of times now that my inbox would shrink in message count without my explicit doing. Generally, your inbox automatically emptying should be conceived a good thing, but it isn’t always. It took me a while to put together the pieces:

• left over messages are all new or old, meaning “not read”,
• offlineimap logs suggested that I deleted those messages on one of my workstations, before offlineimap synchronised those changes to the other machines, and
• it seems to coincide with instances of use of Mozilla Thunderbird and maybe other IMAP clients.

I think I unvealed the mystery: some IMAP clients automatically mark read messages as deleted. Don’t ask me why, I did not configure it, and even though I told Thunderbird specifically not to do it, I have no other explanation than to assume that it doesn’t care about what I want, but marks them for deletion anyway. Firefox decides to block cookies several times a day, despite my explicit requests to store them, and the two are from the same project, so it seems plausible.

Once marked for deletion (by way of an IMAP flag), offlineimap propagates the flag to all clients. Since I set delete=yes for mutt, if I then open and close a mailbox with such messages without noticing them, the messages are purged.

I gave up fighting and solved the problem at a different point, namely mutt (which was doing the deleting anyway):

folder-hook . push '<undelete-pattern>~D<enter>'

Since mutt deletes mail marked for deletion when I close a mailbox, finding those messages at time of mailbox opening must mean that they have been marked outside of the mailer — I use mutt for everything, exclusively. So let’s undelete them.

I can’t see any negative consequences of the above hook.

NP: Oceansize: Efflorescence

I am playing with the thought of using IMAP to send email from my mobile clients. The way this works is that mail is actually placed into a Maildir locally and synchronised with the IMAP server using a tool such as offlineimap. The Courier IMAP server supports such an outbox feature, for other servers you can use inotify or a cron job (I am planning to publish a writeup about my new mail setup sometime soon, so no scripts linked here yet).

Anyway, it’s trivial to get it working, and Dann Frazier has written it all up. The question is more whether I want to, or not. I am soliciting feedback.

The advantages of using an IMAP-synchronised outbox are:

• The ability to review all mail to be sent while still offline. Using mutt, you can even edit them after “sending” but before they are actually sent.

  Undoubtedly, doing this often should be a hint to take more time before sending a message. However, at times there are good reasons why one might want to change a message which was composed e.g. on the plane (or not send it at all), and having to remove the message from the MTA queue and recreate it with the desired changes is unnecessarily time-intensive.

• Wherever you may roam, you can send mail even if port 25 is blocked (provided that the IMAP or SSH ports are open).

• All your mail is sent by a single host, so you don’t have to worry about blacklisted IPs or disclosing information about your whereabouts.

• It’s trivial to post-process mail, including training spamfilters with it (as ham) through a server-side pickup script, as well as e.g. checking for missing attachments locally (you need a local sendmail script for reasons documented here).

Any solution with advantages comes with disadvantages, and these are:

• Added complexity and brittleness: having mutt pipe mail to a custom sendmail-replacement script, which puts a message in place for synchronisation with offlineimap, and a server-side cron-job calling another custom script which pipes the mail(s) to be sent to the real sendmail binary all just seems a little less trustworthy than good ol’ postfix delivering mail via a TLS-encrypted, client-cert or SASL-authenticated SMTP channel to a known mail relay.

• It’s non-trivial to get meaningful notifications in case delivery of a message failed.

• Messages are only sent after IMAP synchronisation carried the mails to the server and the server-side pickup process sent them on. In my setup, this could introduce a delay of 30 minutes even when online, which may be rather undesirable in urgent matters. On the other hand, such delays could easily serve to reduce stress levels (I realise that’s not a problem with a technical solution though).

Can you think of any other advantages or disadvantages? Do you use an IMAP-synchronised outbox and would like to share your experiences with me? Or do you have yet another solution? Write to me!

NP: OSI: Free