5x12 pentomino tiling
«

Yahoo!, DMARC and List Servers

»

As you may know we host lots of mailing lists at The Hug using mailman, which is pretty much the de facto standard open source solution for such things, and yesterday I saw a bounce from Yahoo! which had rejected an email sent through the list server to a subscriber using a yahoo.co.uk email address. The reason took my breath away.

The page the bounce points you to explains that:

Your message wasn't delivered because Yahoo was unable to verify that it came from a legitimate email sender.

and goes on to mention methods such as DKIM signing which I've talked about here before. Which was odd as we do DKIM sign all our outgoing mail, including mail from mailing lists. But here's the rub: they also use DMARC and the email originally came from someone at an aol.com address, who do too.

The page says of mailing lists:

In almost all cases, we recommend that you switch to sending mail from your own domain. For mailing lists, also known as "listservs," you should change your sending behavior by adding the mailing lists' address to the "From:" line, rather than the sender's address. Also, enter the actual user/sender address into the "Reply-To:" line.

The DMARC web site gives similar advice, note option 3. in particular, which is where we would need to go as we modify the Subject: header and the message body.

So let's be clear what they're saying here. Suppose I mail milton-chat (a mailing list we host). Then currently the email as sent to the list will have three headers as follows:

To: milton-chat@the-hug.net
From: paul@the-hug.org
Reply-To: milton-chat@the-hug.net

Now I'm aware there is a element of religion here: some people believe you should set the Reply-To: header to the sender, then people can click on Reply to reply only to the sender and Reply All to reply to the list. All I can say is that in my experience that doesn't work for many users - they simply don't understand the distinction - and we've found working the way we do, which is the old fashioned way mailing lists worked, to be far easier for people to grasp.

Anyway what they're saying is that we should change this so that instead the mail sent to the list reads:

To: milton-chat@the-hug.net
From: milton-chat@the-hug.net
Reply-To: paul@the-hug.org

Which works fine if you already operate on the basis I described above where Reply-To: is set to the sender but for us that would mean re-training subscribers in how to use all our lists. So we would have to use:

To: milton-chat@the-hug.net
From: milton-chat@the-hug.net

And you can see the problem right away: the sender's identity is now lost.

I discover having Googled about that I'm not the first person to come across this. In April The Register covered the issue and it also looks like the AOL is doing the same thing. Their page has pretty much exactly the same wording as Yahoo! in their advice to list owners.

Back in April the main issue was list servers not correctly DKIM signing their emails. Well we do that, so we were safe from that, but now Yahoo! have moved it up a notch and they're respecting AOL's p=reject DMARC record to reject mail apparently from an AOL user (which it was of course) but DKIM signed by our list server rather than AOL's.

So I'm now left with three options:

  1. Change the From: address to be the mailing list address, set the Reply-To: address to be the sender and then re-educate all our subscribers.
  2. Change the From: address to be the mailing list address and put the sender's email address at the top of the message body.
  3. Ignore the issue and accept that it will become impossible for people with email addresses at Yahoo!, AOL and an increasing number of other major ISPs to use our list server.

Sadly I don't think option 3. is tenable, much as I would like to go that way. At the moment I'm leaning towards 2. as the grief of re-educating subscribers on option 1. both initially and repeatedly thereafter doesn't bear thinking about.

Anyway I'd be interested in people's thoughts on this one, either using the "Comment on this article" link below or by mail to


Update 10/09/14

So, as you'll see from the comments below, it looks like option 2. but modified by Steven's suggestion is the best solution. So this means that the email to the list would look like this:

To: milton-chat@the-hug.net
From: "Paul Oldham paul@the-hug.org" <milton-chat@the-hug.net>

That actually renders quite well in a couple of mail clients I've tried it in (Thunderbird and SquirrelMail). You can still clearly see the original sender and their email address as most mail clients now tend to preferentially display the text version of the From address. It'll probably screw up automatically updated address books mind you.


Update 24/09/14

As Dave pointed out below later versions of mailman do resolve this problem. Sort of. We've got 2.1.6 on the new server and this is what it does:

To: milton-chat@the-hug.net
From: "Paul Oldham via milton-chat" <milton-chat@the-hug.net>
Reply-To: paul@the-hug.org

Which is fine as far as it goes but if, like we do on our lists, we don't want Reply-To: set to the poster then what you end up with is this:

To: milton-chat@the-hug.net
From: "Paul Oldham via milton-chat" <milton-chat@the-hug.net>

So other posters can't see the email address of the poster. After some head scratching I've found a solution though. We already pre-process the email before passing it to mailman so as part of that I've change the From: address to "Paul Oldham paul@the-hug.org" <paul@the-hug.org> so then mailman produces a mail that looks like this:

To: milton-chat@the-hug.net
From: "Paul Oldham paul@the-hug.org via milton-chat" <milton-chat@the-hug.net>

Which is pretty much the result I was looking for.

Tags: internet, linux Written 09/09/14


Previous comments about this article:

On 09/09/14 at 11:25am Paul wrote:

Steven, via Twitter, tells me that "Yahoo Groups now does 2 and puts sender's address in the free-text bit of the 'From' field." and that sounds like a better option than putting it in the message body.

On 09/09/14 at 12:29pm Mary-Ann Johnson wrote:

I'd go for (2), as modified by Steven's suggestion. However, I think you'd also have to change your archive script so that it anonymises that email address, in the way the it currently does for the "From:" field.

On 09/09/14 at 12:52pm Dave Holland wrote:

The latest version of Mailman has the ability to identify messages that originate from "problem" (DMARCed) domains, and either munge the from address or MIME-wrap the entire message. Is that any good?

On my own lists - not using Mailman - I went for option 3, but the few Yahoo subscribers I had were amenable to changing mail provider(!)

On 09/09/14 at 4:50pm Kieran Cooper wrote:

The only other option I could find was to set up some kind of 'mail replacement' where you assign each author an address on your domain which you can then use as a From address. When replies hit your server, you redirect them to the original recipient. This is obviously not trivial so I don't know if it's feasible - or indeed if anyone is writing it for MailMan.

I absolutely agree that it will be very hard to re-educate people not to be able to reply direct to an original sender. Option 2 would also mean all sorts of rubbish (out of office replies etc) directed at the list posting address so there will be more work needed on moderation...

Comment on this article

«
You can follow these posts on Twitter at @Wibblings
»
I am currently reading:

Scotland: The Autobiography by Rosemary Goring Bring on the Empty Horses by David Niven

(?)
Word of the Day:
carcade