I’ve had to book a few flights with BA recently. Up until a couple of weeks ago their acknowledgment e-mails came through fine. And then I stopped receiving them. Taking the time to delve in to the mail logs yesterday I noticed this:

Aug 20 07:47:45 jc sm-mta[15347]: l7KElEmk015347: ruleset=check_mail, 
  arg1=website +LHS=RHS@bounce.baplc.com, 
  relay=ceba-mgw04.baplc.com [163.166.43.64], 
  reject=553 5.1.8 website +LHS=RHS@bounce.baplc.com... 
  Domain of sender address website+LHS=RHS@bounce.baplc.com does not exist

(I’ve redacted the left and right hand side of the actual e-mail address it was being sent to)

If that’s just so much gibberish to you, it says that BA are sending e-mails with a return path of …@bounce.baplc.com. Working through the logs shows that they’ve been doing this for some time.

But at some point in the last few weeks, someone at BA has removed the bounce.baplc.com entry from their DNS. So my, and countless other systems around the world, will begin rejecting messages.

This rejection is quite correct. Since bounce.baplc.com doesn’t exist, my system (and any other system with the same configuration) will have nowhere to send any bounces that might occur. And sending messages from domains that do not exist is also an exceedingly common spammer tactic.

I’ve used the “Report problems with our site” feature to report this to BA, but I don’t have high hopes of anyone listening.

Share This

Apparently (and this is second-hand, so treat it with some caution), the Korean Information Security Agency (KISA) is producing an RBL of domains to blacklist. This is complemented by a whitelist, called WhiteDomain, seeded by using published SPF records.

There’s a webpage for the KISA RBL, with both English and Korean versions. It looks like there’s more content on the Korean version than on the English version, but most of it’s text on images which Google can’t translate, and my Korean’s not up to much.

Many Korean ISPs are using this RBL and whitelist

However, the process for seeding WhiteDomain is only correctly handling SPF records that use the ip4: mechanism. Other mechanisms (like a:, ptr:, and mx:) are being ignored. So if your SPF record doesn’t use the ip4: mechanism (and there are many legitimate reasons for not doing so) you are going to find your e-mail blacklisted by many Korean ISPs.

This is broken for a couple of reasons. First, the SPF specification is quite clear about what a conforming SPF implementation must do:

4. The check_host() Function

The check_host() function fetches SPF records, parses them, and interprets them to determine whether a particular host is or is not permitted to send mail with a given identity. Mail receivers that perform this check MUST correctly evaluate the check_host() function as described here.

An implemenation that claims to process SPF, but that does not follow parts of the specification is broken, pure and simple.

Second, there’s nothing preventing spammers from publishing their own SPF records, authorizing their mail relay. If WhiteDomain is being seeded purely by SPF (and I don’t know for certain that it is, so take this with an appropriately sized piece of salt) it’s likely to become less and less useful over time.

Share This

Sendmail has supported pausing before displaying the greeting as an anti-spam measure since 8.13.0.

To use it, add a line like:

FEATURE(`greet_pause', `10000')

to your sendmail.mc file, and rebuild the sendmail.cf file. The delay is specified in milliseconds, so that example will cause Sendmail to delay for 10 seconds before displaying the greeting banner.

What that feature was introduced, Sendmail would log connections that fell foul of this check with this line:

rejecting commands from host [ip-addr] due to pre-greeting traffic

This is helpful in determining how many times the check fired (and therefore how many connections it’s stopping). The hostname and IP address information can also be mined from the logs and used to update any blacklists or firewall rules you’re maintaining — if the site is going to trigger this block then you may as well prevent from connecting, and save some connection slots.

However, it doesn’t tell give you any idea of what would happen if you were to increase or decrease the greeting pause time. Suppose you have the delay set to 10 seconds. It may be the case that all the violators are trying to send data within the first second. In which case a 10 second delay is going to penalise all connections for an unnecessary 9 seconds.

On the other hand, it might be the case that blocks because of a 10 second greet pause delay are spread out over the entire 10 seconds. In which case there isn’t much opportunity to reduce the delay, and it could be argued that increasing the delay makes sense.

To be able to answer questions like this I sent in a patch, and from Sendmail 8.14.0 (or 8.13.5 if you enabled the _FFR_LOG_GREET_PAUSE feature) Sendmail now reports:

rejecting commands from host [ip-addr] due to pre-greeting traffic after num seconds

This is sufficient information to produce a histogram that shows the frequency of the delay at the time that connections are rejected because they triggered this check. Here’s an example of what such a report might look like.

# == 1 connection
 Delay   Count          Histogram
     0  (    12)        ############
     1  (     9)        #########
     2  (    14)        ##############
     3  (     9)        #########
     4  (    11)        ###########
     5  (    27)        ###########################
     6  (     8)        ########
     7  (    13)        #############
     8  (     6)        ######
     9  (    11)        ###########

In this example it’s clear that the pre-greeting checks are firing evenly across the range of delays — with the exception at 5 seconds. There’s a definite spike there. That could be inidicative of a particular popular piece of spam-ware that’s configured to only wait 5 seconds before sending mail.

Here’s the Perl script that produced that report. Happy spam hunting.

#!/usr/bin/perl
 
use strict;
use warnings;
 
use List::Util qw(max);
 
# Read maillog from stdin, produce histogram showing greetpause
# distribution
 
my %pauses    = ();
my $max_count = 0;
 
while(<>) {
    next unless $_ =~ /pre-greeting traffic after (\d+) seconds/;
    $pauses{$1}++;
    $max_count = max($pauses{$1}, $max_count);
}
 
my $scale = int($max_count / 40) + 1;
 
print "# == $scale connection", $scale == 1 ? '' : 's', "n";
 
print " Delay\t Count\t\tHistogramn";
foreach my $delay (sort keys %pauses) {
    printf "%6d\t(%6d)\t%s\n", $delay, $pauses{$delay},
        '#' x ($pauses{$delay} / $scale);
}

Share This

First, the new HeloName option.

When Sendmail needs to deliver mail to a different server, it connects to the remote server, waits for it to send the greeting banner, and then sends:

HELO the.host.name

(or EHLO the.host.name, but I’m keeping this simple). Sendmail uses the local, fully qualified hostname of the host when it does this. So if you host is called foo.example.com Sendmail will send HELO foo.example.com.

What happens next depends on how the remote system is configured. It is very typical for the remote site to do a DNS lookup on the provided hostname, to make sure that it exists. This is a common anti-spam measure. If the given hostname doesn’t exist then the rest of the connection will be rejected.

Further, it’s also common to do the reverse lookup. So if foo.example.com resolves to the IP address 1.2.3.4, the remote site will then look up 4.3.2.1.in-addr.arpa, and verify that that resolves back to foo.example.com. Again, if it doesn’t, the connection is refused. This is a less common anti-spam technique, but it’s still found in regular use at hundreds of thousands of sites around the world.

You’ve probably worked out how this can cause problems. It’s common, especially in corporate environments, for hosts to have multiple hostnames, especially if they are internet facing hosts. There’ll be one name by which the host is known internally (and is only published in the internal, corporate DNS), and one name (and associated IP address) by which the host is known externally.

This can lead to situations where the mail server connects to the remote server, and sends:

HELO my.internal.name

my.internal.name is not published in the public DNS, so the remote server rejects the rest of the connection.

It may be even more complicated than that — for example, the internet facing mail relay might be behind a firewall that carries out Network Address Translation (NAT) — in which case, even if the server is configured to use the correct hostname in the HELO step, that hostname might not resolve back to the right IP address, and again, the connection may be refused.

Prior to the HeloName option being added to Sendmail you could use the confDOMAIN_NAME configuration option to specify the full domain name to use.

This works, but has other side-effects. In particular, the value of confDOMAIN_NAME appears in Received: headers, and if you set it obscures the actual hostname, which can make analysis of messages through the Received: headers more difficult.

This is the situation I ran in to at $work. We have many Internet facing mail servers relaying mail to the outside world. Each server knows its own internal hostname, but has no knowledge of the hostname it presents to the outside world, nor does it know the IP address that the packets it sends will finally have (all the servers are behind NAT devices).

I considered configuring each server with this information, but decided that that would be a bad idea. It would have meant tightly coupling our configuration information with configuration information maintained by an entirely separate group (the Networks Team). Should they decide to renumber a NAT’d IP address they would need to ensure that we were informed, and that the change was co-ordinated.

That’s extra work, extra complexity, and extra opportunity for things to go wrong.

Each server’s IP address is already in a large A record, mail.example.com. I considered adjusting confDOMAIN_NAME and related settings so that each outbound mail relay considered itself to be mail.example.com, but rejected that idea on the previously mentioned issues relating to Received: lines and the extra complexity it introduces when debugging.

So, plan C. Specifically, I wrote a small patch for Sendmail that introduces a new configuration option, HeloName. If set, Sendmail will use the value of this option when it sends HELO/EHLO, instead of the fully qualified hostname (or confDOMAIN_NAME setting).

Our Sendmail configuration files now contain this directive:

define(`confHELO_NAME', `mail.example.com')

This is sufficient for all our outbound mail relays to HELO with the same name. This name contains all their IP addresses. And the Received: headers still contain the correct internal hostnames, to make troubleshooting easier.

If we’d been using a closed source MTA this wouldn’t have been possible.

Share This

1. Send spam
2. Host phishing sites
3. Carry out denial of service attacks

There are many DNS based Black Lists (DNSBLs) operated by numerous groups that aim to list the IP addresses of systems that have been compromised in this way. This allows mail server operators to configure their systems to query these DNSBLs and reject messages that are being sent from a compromised system.

The problem with this approach is that it’s not visible to the owner of the compromised system. They might notice that it’s behaving a little slower, or that their network connection doesn’t seem as fast as it was, but they’re not going to know why, because there’s no easy mechanism to alert them.

In a perfect world, Internet Service Providers would monitor these DNSBLs, notice when IP addresses of their customers appear on them, and terminate (or provide limited) service to that customer, along with appropriate assistance to help them clean their system.

In practice this rarely happens.

It occured to me that one way to make the fact that their PC has been compromised more visible to end users would be by enlisting the help of companies that host or provide online game environments.

For example, consider Second Life, Eve, or World of Warcraft. All huge, multi-player games/environments, to which millions of people connect every day.

If the companies that host these environments were to check the IP addresses of connecting systems against these DNSBLs, they could provide a warning to the player that it’s highly likely that their PC has been compromised, and that they should make sure their anti-virus is up to date, and so on.

Further, suppose you’ve got a PC and an XBox or Nintendo Wii at home¹. Both of those game systems support online play. And through a networking quirk that I don’t need to go in to here (NAT), it’s highly likely that the PC and the games console(s) are going to appear online with the same IP address.

So if the PC appears on a DNSBL the Wii or XBox is going to appear on that DNSBL too. This provides an opportunity for Microsoft and Nintendo to check the IP address, and again, place a warning in the “dashboard” (XBox dashboard, Nintendo Wi-Fi Connection) that their systems display to the user as they go online.

This could significantly raise the awareness of owners/operators of compromised PCs.

¹ Guess what I got for Christmas :-)

Share This

But not, perhaps, in the way that you might imagine.

As already chronicled, some of my writing recently made it in to The Guardian. As is the way of these things The Guardian like to pay their writers, so I sent off my details to their billing department and waited for the money to come rolling in (as you do).

It turns out that, by an odd coincidence, I’m not the only Nik Clayton to write for The Guardian. I’m not even the first. This other Nick Clayton (note the extra “c”) has written a number of columns for them, and they’re also about technology matters.

This much became apparent when I received an e-mail from The Guardian’s billing department today confirming that they had dispatched payment for two articles that Nick had written to me. This e-mail contained Nick’s name and address details, and the payment details (amounts) for the articles he’s written. But it also contains my bank details (account number and sort code). The money hasn’t been deposited in to my account yet, but I imagine it soon will be.

A bit of Googling turned up Nick’s site, and a bit more Googling turned up a phone number, so I’ve called him, and had the slightly surreal experience of:

Good evening. Could I speak to Nick Clayton?

Speaking

Hi. It’s Nik Clayton here…

Now I know how Dave Gorman must feel.

I’ve tried calling The Guardian’s billing department but the number given in the e-mail redirects to voice mail at the moment, so I’ll be in touch with them again tomorrow morning.

There are at least four risks here.

First, The Guardian’s billing department will apparently change the sort code, bank account, and e-mail address details that they hold for writers on the basis of a single unauthenticated e-mail. My message to them was:

Charles Arthur asked me to send my payment details for

http://technology.guardian.co.uk/online/insideit/story/0,,1954392,00.html

to you.

Sort code is XX XX XX, the account number is XXXXXXXX.

Please let me know if there are any problems.

Second, when they pay their writers they send out an e-mail that contains, in clear, the writer’s name, reference number, full address, sort code, bank account number, and the values of the payments. This may well be enough to carry out a social engineering attack.

Third, this could easily have gone the other way, and my bank account details could have been forwarded to Nick Clayton. Had he been nefarious I imagine that (given that we share the same name) these could have been used to carry out a very effective identity theft.

Fourth, had I not been quite so honest I could probably have got away with this for some time — at the very least, continuing to earn interest on the money that The Guardian have paid.

Hmm. I wonder if The Guardian would like to use this as the basis for an article…

Share This

The first concerns what happens if you’re hosted at an ISP with an anti-spam policy, an itchy-trigger finger, and a support desk that is devoid of clue.

It appears as though the fine folk over at The Weekly had their infrastructure on a shared server at their ISP, HostingPlex. That same server was then used by a spammer to send spam, which was caught by SpamCop. Rather than track down the actual culprits, HostingPlex have locked The Weekly’s account, and are demanding US$150 to reinstate access to the server, while ignoring repeated e-mails from The Weekly that contain what appears to be pretty straightforward evidence of their innocence.

I paraphase somewhat, you can read the The Weekly’s side of the story for yourself.

In other news, “Thoughts on stopping spam” appeared, somewhat edited, as an article in The Guardian yesterday.

Update: I hadn’t realised that this piece had made it in to the print edition as well as on the website. Photographic proof (largely for my mother) below.

Share This

There are a couple of other possibilities. The first is that the CAPTCHA system he’s using might be compromised. Some OCR systems can be surprisingly effective on them.

The second is his CAPTCHAs are being reproduced on another site for humans to solve. The canonical example would be where a visitor to a porn site is shown a CAPTCHA and asked to solve it before they can, er, continue. Unbeknownst to them, however, the CAPTCHA is actually coming from Charles’ system, and the solution is then used to send him spam. This is “CAPTCHA farming”.

Searching for “CAPTCHA porn” turns up a number of stories about this over the past few years.

Share This

I’ve fired an e-mail off, which I reproduce below, in the hope that it might be useful to a wider audience.

I see Robbie’s introduced me. In my $dayjob I run the anti-spam engineering for a very large (200,000+ users) financial institution.

There are a few things that I’d like to comment on in your article, http://technology.guardian.co.uk/weekly/story/0,,1948268,00.html. I realise some of these are quotes from other people.

Oh, and in the spam-fighting community there’s the notion of the FUSSP — that’s the Final Ultimate Solution to the Spam Problem. So called because periodically people pop up proclaiming that spam is now a solved problem, if only everybody would adopt their scheme. There’s a taxonomoy of these at

http://www.rhyolite.com/anti-spam/you-might-be.html

On “challenge-response” systems. These are a phenomenally bad idea. Primarily this is because on most spam the From: address is forged. So you receive a spam, and end up sending a challenge to the wrong person. In effect, you are spamming them with your challenges. This doesn’t help.

Even if spam didn’t forge addresses, by using a C-R system you are essentially out-sourcing your anti-spam operation to a myriad of third parties. Do you trust all those third parties to reliably accept the challenge? I know a number of people who will automatically junk any challenges they receive, and, similarly, a number of people who will automatically reply to any challenge they receive, irrespective of whether or not they sent the original message.

Oh, and there’s also the issue of C-R deadlock. Support you and I both run a C-R system. I send you a message, and don’t whitelist you. You send me a challenge. I send you a challenge in response to your challenge… and neither one is ever responded to.

On ISP’s blocking other port 25 mail — this is a very appropriate action to carry out. However, it needs to be carried out in conjunction with a second step — that is, accepting mail submission on port 587. This is a standard port assignment, designed for exactly this situation. Submission on port 25 and 587 are very similar — the difference is that port 587 submission is *designed* to require authentication.

So the big, ISP to ISP type communication continues on port 25, and individuals, such as Philip Parker, would configure their mail program to send mail to their university using port 587, authenticating with his university’s mail servers.

This has still to catch on in a big way, although some of the larger ISPs are moving to support it (in the same way that port 25 blocking is still in relative infancy).

On limiting e-mails sent per day — your correspondents are correct that no single limit is going to be appropriate for everybody. However, two modifications make this more acceptable.

First, provide a mechanism for customers to change their limit, possibly subject to some review. Start them off with a low limit, let them increase it to some capped value if necessary.

Second, change the granularity, so that instead of messages per day it becomes messages per hour, or similar.

On writing a worm that kills botnets. Hopefully the legal ramifications of this are obvious, and don’t need to be stated.

What is missing, though, is the notion that ISPs might hold their customers to greater account. If your contract with the ISP allowed them to levy a charge for any damage to their network or reputation caused by your failure to keep your computer botnet free then customers might start to realise the true cost, and invest more time and attention in keeping their systems clean.

The “Adopt IPv6″ comments seem misguided. IPv6 helps solve many problems (and Marshall’s comments in the article are quite correct) but it will not significantly help in the fight against spam.

Something that wasn’t covered is better detection of abuse by ISPs, with more help for cleaning customers who have become infected.

Botnet traffic patterns are relatively obvious (e.g., a customer host that’s gone from sending 20 e-mails a day to 20,000). When this is detected the majority of the customer’s internet access should be shut down. Attempts by them to browse any web site should redirect them to a site at the ISP that explains why their access has been curtailed, and allows them to download tools that can help them clean their PC.

Obviously, this needs to be backed up with a mechanism to rapidly unlock customers who are caught because their traffic patterns have changed legitimately, but those should be few and far between.

Finally, to respond to your comment to Robbie:

Charles Arthur wrote:
>> thanks for your letter. Interesting, though it seems to me that
>> increasing the time taken for mail transfer from a few milliseconds
>> to several hours would have very serious effects; and you’d have to
>> be certain you were tarpitting the right IPs.
>>
>> Most of all, it wouldn’t stop spam from botnets, which are
>> individual PCs.

I assume Robbie’s talking about delaying all inbound connections.

Broadly, there are three ways to do this.

If you’re a public spirited individual, and you have a lot of resources at your disposal, you may use various public lists of spamming IP addresses, and wait for connections from those IP addresses. Then you just respond to them v-e-r-y s-l-o-w-l-y.

Surprisingly, that does tend to work, at least a little, as it results in the spammers needing larger and larger botnets to send the same amount of spam. Sadly, it seems that acquiring larger and larger botnets is not proving to be too much of a problem, so this is, at best, a delaying tactic.

You may choose to adopt what’s called “greylisting”. Essentially, you have your servers maintain a record of IP address/sender address that you see.

If you receive a connection from an IP address and sender that you’ve not previously heard from, you temporarily reject the message, and note that you’ve done so.

If you receive a connection from an IP address and sender that you’ve previously heard from, you accept the message (or, at least, allow it to go through to the next stage of the spam filtering).

The thinking here is that most spam-sending software, when it receives a temporary rejection, will treat that as a permanent failure, and move on.

Well behaved, non-spam sending software will, on the other hand, hang on the message, and try to send it to you again at some point in the near future.

This is, as you noted, introducing a deliberate delay in to the mail flow, which may not be acceptable. Note, though, that you only do this once per IP/sender address combination, so it’s only the first message that is ever delayed.

The third way to do this is to insert delays at the protocol level, taking advantage of the fact that much spam sending software tries to get spam out as quickly as possible, and will ignore any deliberate delays you introduce. These delays, which may be of the order of a few seconds, or tens of seconds, don’t introduce significant delay in to the mail flow for legitimate senders, but are much more damaging to botnets.

I hope all that’s helpful. Please don’t hesitate to get in touch if you’ve got any other spam related (or general e-mail) queries.

N

Share This