Showing posts with label OpenBSD spamd. Show all posts
Showing posts with label OpenBSD spamd. Show all posts

Friday, May 15, 2026

OpenSMTPD Is The Mail Server For The Future

OpenSMTPD profile image, see https://www.opensmtpd.org/
Image credit: the OpenSMTPD project

© 2026 Peter N. M. Hansteen

The SMTP mail server for the 21st century and onwards is OpenSMTPD, which is developed as an integral part of OpenBSD, but available in a portable variety too.

It was one of those things that I had fully intended to do years ago, but I only got around to actually doing once there was a definite deadline to get it done.

The time has come, as OpenBSD 7.9 will leave the exim package behind, and exim users will need to find a replacement before upgrading. This article describes my transition to OpenBSD's own OpenSMTPD mail server.


OpenBSD 7.9 will leave the exim package behind, and exim users will need to find a replacement.

OpenSMTPD (smtpd) is in the base system.


When OpenSMTPD was first introduced in the OpenBSD base system in OpenBSD 4.6 in October 2009, I had already been running a mail service for some years.

At the time I still found it convenient to keep using exim as the real mail server, protected by OpenBSD spamd in the incoming signal path and with a combination of spamassassin and clamav for content filtering.

It seemed quite tempting to me to play around with at the new smtpd at the time, but the initial version of the new mail server was not yet considered quite ready for prime time.


Note: This piece is also available without trackers but classic formatting only here.

The pace of development was quite hectic in the early years, and by the time smtpd replaced the classic sendmail as the default mail server in OpenBSD with the November 2014 OpenBSD 5.6 release, I had just completed the third edition of The Book of PF and I was interested, but the writing had been quite a drain on my energy.

And of course, the mail server setups I had running for myself and friends I thought of as complex enough that moving to something else would require quite some preparation and testing. So I would leave looking into the new mail server software properly for another day, soon to come, I was sure.

An Old Setup, Maintained With Much Love and Care

There are some hints of what that setup did (and still does) in the 2012 piece In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe (also tracked, prettified), but the main features are:

  • Two (originally three) separate sites, each with their own domains, where the other site(s) provide secondary MX duty for the other(s), each with a spamd-instrumented OpenBSD machine as the Internet-facing part of the mail setup
  • The OpenBSD machines perform spamd greylisting and greytrapping, but also provide content filtering on behalf of another set of domains with their own, not necessarily Internet-exposed, mail servers that receive the filtered mail relayed to them by the Internet-facing mail services.

This setup, with OpenBSD spamd in a greylisting and greytrapping setup in front and content filtering as the second stage before finally relaying to the protected mail hosts, worked well enough that we simply kept the systems running with only routine system and package upgrades and minor adjustments to configurations as needed.

In short, domains to be served came and went, but the spamd, exim and clamav+spamassassin combination stayed, on the ever reliable OpenBSD platform.

Time To Move On, Wait, Then Finally ...

Over the years, there were several episodes with medium to severe security flaws discovered in the exim codebase, but the OpenBSD package was generally well maintained and fixes tended to appear within a reasonable time.

From time to time OpenBSD developers and port maintainers discussed dropping support and removing exim from the package system, but it was only in early 2026 it finally happened.

OpenBSD 7.9 will ship without an official exim package.

So it was finally time for even this holdout to move to something else.

And Of Course, A False Start

Other OpenBSD users had kept telling me how good OpenSMTPD had become, so I decided now was the time, so I dug out some old notes and started experimenting.

Those old notes turned out to be utterly useless, and for a reason: The OpenSMTPD 6.4 release was the result of a major code overhaul that also changed important parts of the smtpd.conf syntax.

Unfortunately a majority of the third party guides out there that turn up early in search results still use the old syntax, and as a consequence, are useless, at least to users on OpenBSD or other platforms that have kept their code reasonably in sync. A useful rule of thumb is, if you find yourself reading an OpenSMTPD guide that is dated before 2020, do yourself a favor and move on to something newer.


If you find yourself reading an OpenSMTPD guide is dated before 2020, do yourself a favor and move on to something newer.

The Task At Hand: The Analysis

But to the problem at hand. The setup I was setting out to convert, was one that needed to accommodate

  • Inbound mail for users in the local domains, where we are the primary mail exchanger
  • Inbound mail for users in the domains where we are the secondary mail exchanger
  • Inbound mail for the users in the domains where we are the primary public-facing mail exchanger, but where we actually only relay after greylisting and filtering
  • Outbound mail from the local domains
  • Outbound mail from networks we have chosen to trust enough to relay for

The mail exchanger (MX) records for all domains involved were already in place, as was other relevant DNS information such as SPF, DKIM and DMARC records. TLS certificates and a regime for maintaining them was already in place, using LetsEncrypt tools.

That analysis converted to smtpd.conf logic would be:

  • We keep the existing /etc/mail/aliases file, the formats are compatible
  • OpenSMTPD conveniently has tables, which can be either simple lists or key-value pairs. Our tables are:
    • domains_local lists the domains we receive mail for to handle locally
    • relay_for_domains lists the domains we only filter for, then relay to
    • domain_relays is the list of domains and their final destination mail exchangers as list of key-value pairs
    • relay_from_ips is the list of IP addresses and networks we allow relaying for

Now The Actual Implementation

So I set to work from that specification, and at the end of that afternoon I had reached the conclusion that

  • setting up for TLS was the easiest, with simple pki statements
  • listen statements actually do the work in very limited space
  • routing to local delivery and forwarding is easy with a combination of action and match rules
  • for filtering, clamav had not actually been of much use to my users (no Windows users among them), and that the filtering options that made use of spamassassin for the back end were either not functional or I was too dense to make any sense of them.
    I ended up testing a modern alternative, rspamd, which is available via the OpenBSD package system.
  • dkimproxy looked like a good candidate for signing outgoing messages, so I tested that for a while, but on the advice of Martijn van Duren, I switched to dkimsign, which is available on OpenBSD as the package opensmtpd-filter-dkimsign.

In addition to smtpd, which is already in base, this configuration requires the packages opensmtpd-filter-dkimsign and opensmtpd-filter-rspamd.

Installing both via pkg_add have the packages pull in all required dependencies.

With the prerequisites in place,

disable and stop exim
doas rcctl disable exim && doas rcctl stop exim

disable and stop clamav
doas rcctl disable clamav && doas clamav stop clamav

disable and stop spamassassin
doas rcctl disable spamassassin && doas rcctl stop spamassassin

At some point, you should remove the packages with

doas pkg_delete packagename

and follow the steps outlined in the package delete message.

Don't remove the exim configuration, though, until you have copied the useful parts across to your new /etc/mail/smtpd.conf.

I ended up with this configuration (lightly edited for brevity)

---- /etc/mail/smtpd.conf
table aliases file:/etc/mail/aliases

table domains_local {
"bsdly.com",
"bsdly.eu",
"bsdly.net",
"bsdly.no",
"bsdly.org",
"bsdly.se",
"nxdomain.no",
# plus a lot of other domains, elided here for brevity
}

table relay_for_domains {
"nuug.no",
"blug.linux.no"
# again more domains in the real smtpd.conf, left out here
}

table domain_relays {
"nuug.no" = "smtp://mx1.nuug.no",
"blug.linux.no" = "smtp://mail.lamasti.net"
# again more domains in the real smtpd.conf, left out here
}

table relay_from_ips {
127.0.0.1
::1
# The rest are fictional, RFC5737 and RFC3849
192.0.2.0/24
198.51.100.0/24
203.0.113.0/24
2001:DB8::/32
}

filter "rspamd" proc-exec "filter-rspamd"
filter dkimsign_rsa proc-exec "filter-dkimsign -d bsdly.net -s x -k /etc/mail/dkim/private.rsa.key" user _dkimsign group _dkimsign

pki skapet.bsdly.net cert "/etc/mail/certificate.pem"
pki skapet.bsdly.net key "/etc/mail/privkey.pem"

listen on socket

listen on all port 25 tls pki skapet.bsdly.net filter "rspamd"
listen on all port 465 smtps pki skapet.bsdly.net filter "rspamd"
listen on all port submission tls pki skapet.bsdly.net

action "local_mail" mbox alias <aliases>
action "relay_domain" relay domain <domain_relays> filter "rspamd"
action "outbound" relay filter dkimsign_rsa

match from local for local action local_mail
match from any for domain <domains_local> action local_mail
match from any for domain <relay_for_domains> action relay_domain
match from src <relay_from_ips> for any action outbound
match from local for any action outbound
----

The specific domain names and IP addresses will be different for the secondary site, as it will for any configuration you will set up.

After some logs and messages observation, I also ended up with minor modifications to the rspamd config,

---- /etc/rspamd/local.d/actions.conf
reject = 10; # final reject
discard = 15;
add_header = 6; # mark spam
greylist = null; # do not greylist, we have spamd for that

# Custom action (referenced by force_actions), no own threshold
phishing = {
flags = ["no_threshold"];
}
----

That is the entire configuration. With the somewhat longer list of domains and networks, the net length of my configuration now is

$ grep -vc \# /etc/mail/smtpd.conf
104

104 lines, while the previous exim config with comment lines stripped out ran to

$ grep -vc \# /etc/exim/configure
380

380 lines.

The smtpd.conf configuration is readable on par with pf.conf, with similar readability features.

At any time after installing the packages and disabling the previous services, enable and start the new services.

To (re)enable smtpd as the default mail server after running with exim, run

$ doas /usr/local/sbin/exim-disable

to restore /etc/mailer.conf to its original state.

If all else fails, you can easily retrieve a pristine version from the OpenBSD CVS.

To enable the new services, run

$ doas rcctl enable smptpd && doas rcctl start smtpd
$ doas rcctl enable redis && doas rcctl start redis
$ doas rcctl enable rspamd && doas rcctl start rspamd

You should see activity fairly soon by monitoring /var/log/maillog, such as

$ tail -n 500 -f /var/log/maillog

When you are satisfied that mail flows in and out and is relayed where you want it to, it is safe to remove the exim, clamav and spamassassin packages and follow the instructions in the pkg_delete messages to free up some space.

And yes, considerably more complicated configurations are possible, especially in the filtering department.

But I was pleasantly surprised at both how simple the transistion has proven to be and the prospect for having truly maintenance and enhancement friendly setup going forward.

The transition process has showed me that the OpenSMTPD product is solid, like the wider OpenBSD environment. Making OpenSMTPD the default mail server software was no doubt one of those extremely good decisions the OpenBSD project has made, and even latecomers like myself applaud the decision.

OpenSMTPD and OpenBSD both are characterized by their developers' ability to not only learn from earlier iterations of development of the operating system and the mail server component, but also to come up with new, and some times radically different, approaches to known problems that result in a more secure and more useable product. To my mind, this is the mail server and the operating system for the future.

If you are interested in setting up smtpd with more filters or other ones, quite a few are available, including such things as opensmtpd-filter-dnsbl, which pulls in DNS blocklists from the sources you specify.

OpenSMTPD is available on a wide variety of platforms, including various Linux distributions and BSDs such as FreeBSD via its -portable variety.

I have kept this configuration rather minimal, mostly because in my experience, the greylisting and greytrapping spamd is a very efficient and low maintenance outer shield for any mail service. If you are interested greytrapping, the infrequently updated Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (also tracked, prettified) provides more reading material via its numerous links than you could reasonably take in during even a long evening.

If you would rather have a book that covers more networking topics with OpenBSD and FreeBSD as the platform, and includes a fairly extensive treatment of spamd, The Book of PF, now in its fourth edition, is for you.

Good night and good luck!

I want to thank Martijn van Duren for useful advice while working on this article.


OpenSMTPD is the Mail Server for the Future is © 2026 Peter N. M. Hansteen (published 2026-05-15)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Sunday, December 28, 2025

Why 451 is Good for You - Greylisting Perspectives From the Early Noughties

© 2025 Peter N. M. Hansteen

A software vendor was using SMTP spamware to send license keys customers had paid thousands for. A measured rant was in order, and turned out to be quite enlightening.

While looking through directories of old material, I stumbled upon what was most likely the main part of a complaint sent to a software vendor for failing to deliver a license key file the company I worked for then had paid some thousands of dollars for.

The file as I found it was dated August 2010, but was likely a preserved copy of a draft that was written some time before the first edition of The Book of PF (now in its fourth edition) was published, in response to the non-delivery incident. A quick investigation had me conclude from my spamd logs that their side did not play well with greylisting.


Note: This piece is also available without trackers but classic formatting only here.
I have revisited the handling sites that do not play well with greylisting theme a number of times, such as the 2018 piece Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting (also here).

But I found these early notes interesting enough that I include them here, with only minor redactions to protect the (relatively) innocent:

SWCrafters' reaction to finding out that their messages do not get through, essentially blaming "inaccurate spam filtering" was not unexpected, but I will take the opportunity to explain a few things about how Internet email works and how this makes their position at odds with reality.

Even though Internet services are offered with no guarantees, usually described as 'best effort' services, a significant amount of effort has been put into making essential services such as SMTP email transmission fault tolerant, making the 'best effort' one with as close as does not matter to having a perfect record for delivering messages.

The EXECUTIVE SUMMARY of this message is that the matter which trips up the delivery of SWCrafters' license-carrying emails is the fact that their email sending software's best effort at delivery falls significantly short of what current Internet standards require.

The current standard for Internet email transmission is defined in RFC2821, which in section 4.5.4.1, "Sending Strategy", states

"In a typical system, the program that composes a message has some method for requesting immediate attention for a new piece of outgoing mail, while mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender."
and
"The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery."
Contrast this with the application which sends the SWCrafters license information messages, which according to the data I have avaliable opens two SMTP sessions within a second of each other (the time resolution I have in my logs at the moment), apparently discarding the message without delivery afterwards.

RFC2821 goes on to state that

"Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days."
After all, delivering email is a collaborative, best effort thing, and he RFC states clearly that if the site you are trying to send mail to reports it can't receive anything at the moment, it is your DUTY (a MUST requirement) to try again later, after an interval which is long enough that your unfortunate communication partner has had a chance to clear up whatever was the problem.

A sending strategy which relies on every receiver to be receptive at all times, discarding undelivered messages after only one unsuccessful attempt, possibly makes sense if the data you are sending is unimportant or if your intended targets are unlikely read or even want to receive the messages you send.

If on the other hand the data you are sending matter to either you or the intended recipient, it is in everybody's interest that you use the fault tolerance features which a compliant SMTP mail system offers.

To put this in context, you need to remember that the SWCrafters license messages are the result of some SWCrafters customer ordering at least a thousand dollars' worth or software licenses, with no real upper limit on the dollar value of a single message. The system used to send these messages apparently does not understand SMTP status messages, discarding undelivered messages without a trace. Essentially, the system you are using treats the data your customers expect to receive in exchange for thousands of dollars paid as discardable.

The "greylisting" technique which is in use at justgottahave.faith and other sites means that our systems expect any SMTP sender to understand SMTP status codes and to respect "451 temporary error please try again later" messages. The hows and whys are detailed at https://www.greylisting.org/, with a tutorial which contains a 'close enough' description of how it's done at justgottahave to be found at https://home.nuug.no/~peter/pf/, with the particulars starting at https://home.nuug.no/~peter/pf/en/spamd.html.

We do content filtering as well, but this particular application never managed to get its data sent far enough to encounter content filtering until its IP address got whitelisted (listed as 'known good', or if you will, not having to conform to normal criteria).

Greylisting works extremely well, and since it is both standards compliant (essentially insisting on compliance) and simple to implement you should expect it to be deployed at the next site you are trying to send email to.

If I remember correctly, the other side found a way to send the missing license codes with something that did handle SMTP status codes correctly a short time after the mail that included some version of these notes was sent.

I had originally intended to make the URLs in the text here clickable, but changed my mind when I discovered that the current operators of greylisting.org have decided that a large language model (the current iteration of what passes for artificial intelligence needed to be included in the processing. That will perhaps serve as a sign that the world does move on, if not necessarily in useful directions at all times.

If you want to explore the ins and outs of greylisting and the related phenomenon greytrapping, my recent piece Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (also here) is a way to start. For the greylisting part, the notes above capture the main points.


Why 451 is Good for You - Greylisting Perspectives From the Early Noughties is © 2025 Peter N. M. Hansteen (published 2025-12-28)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon (also here). The latest information I have is that physical copies should be ready to ship by the end of January 2026.


Wednesday, April 17, 2024

Fun Facts About the April 2024 Cisco Attack Data

A commendable attack data dump, lightly analyzed.

In the morning hours (CEST) of April 17, 2024, I found in my social media stream a reference to an Ars Technica article titled UNDER SIEGE — Attackers are pummeling networks around the world with millions of login attempts.

NOTE: A version without trackers but “classical” formatting is available here.

Articles about recent or ongoing attacks are not uncommon, but this time I was delighted to see that the report included a link to the actual data, provided by Cisco subsidiary Talos Intelligence.

When I downloaded the data approximately 09:15 CEST, the data consisted of

5243 unique IP addresses
2105 unique user names
71 unique passwords

I was initially a bit annoyed that the each group of data had apparently not been sorted, so I was a bit worried about possible duplicate entries, but closer inspection showed that I had not needed to worry.

Returning readers will be aware that at nxdomain.no (aka bsdly.net) we have been collecting data on attacks and attackers for some time already, as described in Badness, Enumerated by Robots (also available with nicer formatting but with trackers here) and various material linked from that article.

So naturally my impulse was to see whether there was any overlap between the data Cisco provided and the data collected here.

A few quick rounds for sorting and diffing (or very close equivalents), the results were clear:

of the 5243 unique IP addresses, none were in the currently trapped ssh bruteforcers set or the historical pop3 gropers set.
of the 2105 unique user names, a total of 1595 were not already included in the existing spamtraps list


of the 71 unique passwords, 34 or almost half of the total were not already in the spamtraps list.

If the last item made you chuckle, I am not surprised. But I have also observed at various times that bot herders (or possibly bot feeders) have managed to feed the data the wrong way around to their charges.

The biggest surprise here is that there was no overlap in hosts participating in the campaign against the Cisco customers and hosts that had participated in password guessing against my (for all practical purposes) honeypot system.

One possible explanation could be that the attackers here were targeting only specific products, possibly based on previous intelligence gathering. An alternative explanation could be that they were specifically avoiding certain hosts, such as those running the rather security oriented operating system OpenBSD that we use at this site.

The overlap in user names and passwords mistakenly used as user names with previously collected data here is less surprising.

After this very lightweight analysis, I went to the next logical step and added the offending IP addresses to the bruteforcers list and appended a @bsdly.net suffix to the user names and passwords, and added them to the spamtraps list.

I would like to thank Cisco-Talos for sharing the data on this incident freely.

If you found this article useful, interesting or even annoying, I would like to hear from you.

Good night and good luck.


NOTE: If you follow the references in the various other articles, please keep in mind that the command examples there were written from an OpenBSD perspective. Details of command syntax may be different from the implementations on other Unixlikes.

Sunday, December 25, 2022

The Despicable, No Good, Blackmail Campaign Targeting ... Imaginary Friends?

© 2022 Peter Hansteen

Natalia here speaks to our imaginary friend 185.150.184.92

In which we confront the pundits' assumption that the embarrasment-based extortion attempts would grow more “sophisticated and credible” over time with real data.

It's a problem that should not exist. 

It's a scam that's so obvious it should not work.

Yet we still see a stream of reports about people who have actually gone out and bought their first bitcoins (or more likely fractions of one) in order to pay off blackmailers who claim to have in their possesion videos that record the vicim while performing some autoerotic activity and the material they were supposedly viewing while performing that activity.

And occasionally one of those messages actually find their way to some pundit's inbox (like yours truly), and at times some of those pundits will say things like that those messages represent a real problem and will evolve to be ever more sophisticated.

Note: This piece is also available, with more basic formatting but with no trackers, here.

I am here to tell you that

  1. That incriminating video does not exist, and
  2. The pundits who predicted that those scams would evolve to become more sophisticated were wrong.

If you stumbled on this article because one of those messages reached you, it's safe to not read any further and please do ignore the extortion attempt.

I wrote a piece in 2019 The 'sextortion' Scams: The Numbers Show That What We Have Is A Failure Of Education, also available without trackers, where the summary is,

Every time I see one of those messages reach a mailbox that is actually read by one or more persons, I also see delivery attempts for near identical messages aimed at a subset of my now more than three hundred thousand spamtraps, also known imaginary friends.

Over the years since the piece was originally written, I have added several updates — generally when some of this nonsense reaches a mailbox I read — and while I have seen the messages in several languages, no real development beyond some variations in wording has happened.

Whenever one of those things does reach an inbox, my sequence of actions is generally to save the message and add it to the archive, see if the sending IP address has already entered the blocklist that is later exported and add it by hand if not. Then check if the number of trapped addesses has swelled recently by checking the log file from the export script

$ tail -n 96 /var/log/traplistcounts

See if there is a sharp increase since the last blocklist export

$ doas spamdb | grep -c TRAPPED

Then check for related activity in the log

$ tail -n 500 -f /var/log/spamd

Check for the full subject in the same log file

$ grep "You are in really big troubles therefore, you much better read" /var/log/spamd

Then check older, archived logs to see how long this campaign has been going on for

$ zgrep "You are in really big troubles therefore, you much better read" /var/log/spamd.0.gz

This time, the campaign had not gone on for long enough to show traces in the older archive, so I go on to extracting the sending IP addresses

$ grep "You are in really big troubles therefore, you much better read" /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u

Check for activity from one of the extracted addresses

$ grep 183.111.115.4 /var/log/spamd | tee wankstortion/20221123_trapped_183.111.115.4.txt

Extract the sender IP addresses to an environment variable to use in the next oneliner,

$ grep trouble /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u | grep -vc BLACK | tee -a wankstortion/20221123_campaign_ip_addresses.txt

which will record all activity involving those IP addresses since the last log rotation:

$ for foo in $troubles ; do grep $foo /var/log/spamd | tee -a wankstortion/20221123_campaign_log_extract.txt ; done

You will find all those files, along with some earlier samples, and by the time you read this, possibly even newer samples, in the archive.

When something of the sort inboxes, I probably will go on adding to the archive, and if I have time on my hands, also run similar extraction activities as the ones I just described. But unless something unexpected such as actual development in the senders' methods occurs, I will not bother to write about it.

The subject is simply not worth attention past persuading supposed victims to not bother to get bitcoins or spend any they might have to hand. None of my imaginary friends have, and they are just as fine as they were before somebot tried to scam them.

Good night and good luck.


You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

At EuroBSDcon 2025, there will be a Network Management with the OpenBSD Packet Filter Toolset session, a full day tutorial starting at 2025-09-25 10:30 CET. You can register for the conference and tutorial by following the links from the conference Registration and Prices page.

Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon. We are hoping to have physical copies of the book available in time for the conference, and hopefully you will be able to find it in good book stores by then.


Friday, December 9, 2022

Harvesting the Noise While it's Fresh, Revisited

© 2022, 2025 Peter Hansteen A year's worth of logs yields entertaining but unsurprising findings about spammer behavior.
Spam mail, masked but detected, from the archive

Returning readers will be almost painfully aware that here at nxdomain.no (also known as bsdly.net) we host and maintain a blocklist, which in turn is the product of traffic that hits our mail system with attempts at delivery to one or more of the now more than three hundred thousand known bad addresses, also featured at the blocklist home page.

Note: This piece is also available without trackers but only basic formatting here. 

When I first set up the greytrapping back in 2007, the initial spamtraps were non-deliverable addresses in our domains that I had extracted from mail server logs. I won't bore you with the details (which are anyway documented at length in earlier articles), but it was clear from those logs that the domains we hosted back then were more or less continously subject to Joe jobs, as in somebody sending messages with a forged From: field with a made up address in our domains.

After a while I started extracting the potential new spamtraps from the greylist — actually dumping data from there once per hour as part of the script that also generated the exported blocklist. The basic process is described in the July 25 2007 article Harvesting the noise while it's still fresh; SPF found potentially useful (also available trackerless but with links to tracked articles).

Then today it struck me that while that method is useful, by extracting only from the greylist we will only ever collect the address from the initial connections. Any addresses attempted after the miscreants enter the blocklist will simply not be recorded there.

This of course lead to the question: What did we miss?

Fortunately I keep my logs around for a while, the most easily accessible log archive for my main spamd spans a lttle over a year. So I set about with some very basic grep and awk, which netted me this raw list of targeted addresses from the spamd logs.

The list weighs in at a total of 269903 entries, as counted by wc -l.

Some of those addresses are valid, and a small, but actually significant, number are in domains we do not actually serve here, and some entries do not look like mail addresses at all. The stranger ones could be strings encoded in a character set that spamd is not equipped to handle, or could be other binary data that might have been intended to trigger bugs in some of the variants of fully equipped SMTP servers that are out there. Or simply noise of any other kind, including a byproduct of the not very intelligent extraction one-liner I used.

The target addresses in foreign domains I take as a sign that at least some spamming operators mistake a reasonably configured spamd for an open relay, just like they did all those years ago when I started running the greytrapping.

Some things apparently stay the same no matter how the rest of the world has found a way to move forward.

While I did a few other tasks and finally started writing this article, the bulk of the processes that would answer the question posed earlier (What did we miss?) could fortunately run unattended in the background, and after some manual massaging we are left with a results file, with 1530 entries that were none of

  • actually useful deliverable addresses in our domains
  • existing spamtraps

This means of course that the collection of imaginary friends expanded by the same number, and now stands at 304154 entries.

Which I suppose means that harvesting the noise even after a period of aging for refinement can be a good thing.

The entries added represent a wide variety of phenomena. Quite a few seem to be truncated versions of earlier spamtrap entries, and a fair number of the new entries look like they may have descended from artifacts of stupidity such as products of SMTP callbacks. Proving mainly that in mail and spam handling, there appears to be a space still for the less intellectually astute.

With all of this said, the natural followup question is, given the modest net result, was this worth the effort?

Well, the raw output that yielded 269903 entries needed some manual operations in order to weed out the obvious noise (exact time used not recorded), followed by another background task that took, according to time(1)

    real        105m24.220s
    user        73m3.280s
    sys	        29m14.930s
    

which yielded 1577 entries that were pared down to 1530 entries that met the criteria for inclusion in the circle of imaginary friends (also known as spamtraps).

Before this experiment, the spamtraps list numbered 302625, after including the result here, the count stands at 304154, for a gain of less than one percent of the previous total. Again, if you check back at the traplist home page now, the total number is likely to have increased again.

So was it worth the effort? I feel that as an experiment, it was worth doing.

Whether or not it is an experiment that is worth repeating is a question for another day.

If you have opinions on this, I would love to hear from you, in comments, via email or messages on whichever social media brought you the link to this article.

As always, parties interested in studying the data referenced in this article and other pieces I have written are welcome to contact me for arrangements. I can easily dig out more and rawer data than directly referenced here on request.

Stay safe out there.


As a side note, a slightly improved way of extracting useful data about other domains' mail service via SPF records can be found in the November 2018 artice Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting.

That article (naturally) works from the premise that you are running a recent OpenBSD system.


Addendum 2025-01-12

For those so inclined, it is perhaps worth noting that after a bit of pondering some time after writing this piece, I started looking at extracting other items from the spamd logs log entries.

I ended up with extracting the local parts for new spamtraps from the purported sender addreses of entries for trapped delivery attempts some time mid-2024. This made for a significant increase in the number of new imaginary friends, and by the final months of that year I had also started extracting similarly from the string offered by the spam senders as their host name in the EHLO/HELO exchange, which of course swelled the population further.

The effect is clearly to be seen in the file that records the number of spamtraps added per year, updated via trivial scriptery roughly daily.

I hope this article and its addendum helps inspire others in our efforts of green cybercrime prevention by giving the actually intelligent detection methods less work to do.


Addendum some more 2025-01-18

I suppose it had to happen sooner or later, but as commemmorated in this toot, which said

Likely not blogworthy in itself, but #openbsd #spamd aficionados will get a light chuckle from hearing that some scraping and massaging relevant logs had the number of imaginary friends at https://nxdomain.no/~peter/traplist.shtml for our not-friends to play with roll past the one million mark in the early hours of today CET.

The recent update of https://nxdomain.no/~peter/harvesting_the_noise_revisited.html has links to more info. #spam #antispam #greytrapping #blocklists #cybercrime

Yes, that's right, after I turned to extracting vaguely relevant data from logs in order to salt the mine and poison the well further, the number of imaginary friends quickly grew past the one million mark.

And as if this particular Saturday morning was not already quite weird enough for most tastes, somebot produced another remarkable item that I just could not restist tooting about,

And ref previous toot, the 1006089th imaginary friend to join the collection at https://nxdomain.no/~peter/traplist.shtml is, mail.protection.outlook.com@bsdly.net following this sequence: https://nxdomain.no/~peter/blogpix/2025-01_18_johnson@vicglobalintelligence.com_to_mail.protection.outlook.com@bsdly.net.txt

The bots never cease to amaze #openbsd #spamd #greytrapping #antispam #cybercrime

And the two episodes combined proved addendum-worty, at least, see https://nxdomain.no/~peter/harvesting_the_noise_revisited.html

Yes, you read that right: For reasons known only to the bots' herders (if that), the subdomain that houses mail services for a large number of Microsoft customers entered the lexicon of spammers' spanto: addresses. Only to be included at first sight in the herd of imaginary friends I hope will help poison the spammers' data further.

The activity here did of course not stop the bots from keeping on trying. A few minutes after the second addendum here was added and tooted out, my logs showed the following activity from the hosts involved in trying to spam mail.protection.outlook.com@bsdly.net: https://nxdomain.no/~peter/blogpix/2025-01-18_host_targeting_mail.protection.outlook.com@bsdly.net_all_spamd_log_entries.txt. And more likely than not, they will keep trying.

How was the start of your weekend?

Also worth noting is that if you do try to do this at home, please keep in mind that you will neeed to implement a scheme that keeps actually valid addresses in your domains out of the spamtrap pool. Otherwise regrettable episodes may arise.


Friday, February 28, 2020

The 'sextortion' Scams: The Numbers Show That What We Have Is A Failure Of Education

Subject: Your account was under attack! Change your credentials!
From: Melissa <chenbin@jw-hw.com>
To: adnan@bsdly.net

Hello!

I am a hacker who has access to your operating system.

I also have full access to your account.

I've been watching you for a few months now.

The fact is that you were infected with malware through an adult site that you visited.


Did you receive a message phrased more or less like that, which then went on to say that they have a video of you performing an embarrasing activity while visiting an "adult" site, which they will send to all your contacts unless you buy Bitcoin and send to a specific ID?

Note: This piece is also available, with more basic formatting but with no trackers, here.

Note: Due to a recent incident of fatfingering, some log file extracts and other data linked from this piece are no longer readily available. The data may still be partially restorable, and I will attempt to reconstruct data for valid research purposes if there is sufficient interest.

The good news is that the video does not exist. I know this, because neither does our friend Adnan here. Despite that fact, whoever operates the account presenting as Melissa appears to believe that Adnan is indeed a person who can be blackmailed. You're probably safe for now. I will provide more detail later in the article, but first a few dos and don'ts:
  • Whatever some tempting web site tells you in a popup, unless you know what you are doing, do not install software on your devices from any other sources than the official ones. You do not need to install a new video viewer for that site or update your existing one, neither do you need to enter your administrator user name and password along with your credit card details into an unfamiliar-looking dialog box or web form.
     
  • Unless you know what you are doing, stay away from Bitcoin or other cryptocurrencies. If that message is the first you've heard of Bitcoin, you do not know what you are doing, leave it alone. As assets go, there is not much difference between financial derivatives, toxic waste and cryptocurrencies like Bitcoin, in that they should be handled with equal care and only from a distance unless you are in fact an expert in the field.
     
  • If you are not sure about either of the two bullet points before this one, please forget any shame over what you may or may not have done, and contact somebody you trust and who knows the subject better. This may be an adult such as a parent, teacher, social worker or other, a tech-savvy friend, or for that matter law enforcement such as your local police.

The important point is that you are or were about to be the victim of what I consider a very obvious scam, and for no good or even nearly valid reason. You should not need to become the next victim.

And this, dear policy makers and tech heads in general is our problem: A large subset of the general public simply do not know their way around the digital world we created for them to live in. We need to do better.

In that context I find it quite disturbing that people who should know better, such as the Norwegian Center for Information Security, in a recently issued report (also see Digi.no's article (both in Norwegian only, sorry)) predict that the sextortion attacks will become "more sophisticated and credible". Then again at some level they may technically be right, since this kind of activity starts out with a net negative credibility score.

A case in point: Some versions of the scam messages I have been able to study went as far as to claim that the perpetrators had not only had taken control of the target's device, they had even sent that very email message from there. That never happened, of course, and it would have been easy for anybody who had learned to interpret Received: headers to verify that the message was in fact sent from the great elsewhere. Unfortunately the skill of reading email headers is rarely, if ever, taught to ordinary users.

The fact that people do not understand those -- to techies -- obvious facts is a fairly central and burdening problem, and again we need to do better.

Now let me explain. Things get incrementally more technical from here, so if you came here only for the admonitions or practical advice and have no use for the background, feel free to wander off.

I know the message I quoted at the beginning here is a scam because I run my own mail service, and looking at just the logs there just now I see that since the last logs archiving rotation early Saturday morning, more than 3000 attempts at delivery of messages like the one for Adnan happened, aimed at approximately 200 non-existent recipients before my logs tell me they finally tried to deliver one to my primary contact address, never actually landing in any inboxes.

One of the techniques we use to weed out unwanted incoming mail is to maintain and publish a list of known bad and invalid email addresses in our domains. These known bad addresses have then in ways unknown (at least not known to us in any detail) made it into the list of addresses sold to spammers, and we at the receiving end can use the bad addresses as triggers to block traffic from the sending hosts (If you are interested, you can read elsewhere on this blog for details on how we do this, look for tags such as greylisting, greytrapping or antispam).

If it was not clear earlier, those numbers tell us something about the messages at hand. It should be fairly obvious that compromising videos of non-existent users could not, in fact, exist.

Looking back in archived logs from the same system I see that a variant of this message started appearing in late January 2018. The specifics of that message sequence will be interesting to revisit when the full history of sextortion (I still do not like the term, but my preferred alterantive is at risk of being filtered out by polite society-serving robots) will be written, but let us rather turn to the more recent data, as in data recorded earlier this week.

Mainly because I found the media coverage of the "sextortion" phenomenon generally uninformed and somewhat annoying, I had been been mulling writing an article about it for a while, but I was still looking for a productive angle when on Wednesday evening I noticed a slight swelling in the number of greytrapped hosts. A glance at my spamd log seemed to indicate that at least one of the delivery attempts had a line like

       I am a hacker who has access to your operating system.

Which was actually just what I had been pondering writing about.  

So I set about for a little research. I greped (searched) in my yet-unrotated spamd logs for the word hacker, which yielded lots of lines of the type

Feb 22 04:04:35 skapet spamd[8716]: 89.22.104.47: Body: I am a hacker who has access to your operating system.
Feb 22 04:17:04 skapet spamd[8716]: 5.79.23.92: Body: I am a hacker who has access to your operating system.
Feb 22 04:34:03 skapet spamd[8716]: 153.120.146.199: Body: I am a hacker who has access to your operating system.
Feb 22 04:40:30 skapet spamd[8716]: 45.181.93.45: Body: I am a hacker who has access to your operating system.
Feb 22 04:55:04 skapet spamd[8716]: 93.186.247.18: Body: I am a hacker who has access to your operating system.
Feb 22 05:09:39 skapet spamd[8716]: 123.51.190.154: Body: I am a hacker who has access to your operating system.
Feb 22 05:13:22 skapet spamd[8716]: 212.52.131.4: Body: I am a hacker who has access to your operating system.
Feb 22 05:38:02 skapet spamd[8716]: 5.79.23.92: Body: I am a hacker who has access to your operating system.
Feb 22 05:44:39 skapet spamd[8716]: 123.51.190.154: Body: I am a hacker who has access to your operating system.
Feb 22 06:00:30 skapet spamd[8716]: 45.181.93.45: Body: I am a hacker who has access to your operating system.

(the full result has been preserved here). Extracting the source addresses gave a list of 198 IP addresses (preserved here).

Extracting the To: addresses from the fuller listing yielded 192 unique email addresses (preserved here). Looking at the extracted target email addresses yielded some interesting insights:

1) The target email addresses were not exclusively in the domains my system actually serves, and

2) Some ways down the list of target email addresses, my own primary address turns up.

Of course 2) made me look a little closer, and only one IP address in the extract had tried delivery to my email address.

A further grep on that IP address turned up this result.

There are really no surprises to be had here, at least to a large subset of my supposed readers. The sender had first tried to deliver one of the sexstortion video messages to one of the by now more than quarter million spamtraps, and its IP address was still blacklisted by the time it finally tried delivery to a potentially deliverable address.

Doing a few spot checks on the sender IP addresses in recent and less recent logs it looks like the only two things could be mildly exciting about those messages. One is the degree the content was intended to be embarrasing to the recipient. The other is a possible indicator of the campaign's success: Looking back through the logs for the approximate year of known activity, it even looks like the campaign became multilingual, while retaining the word "hacker" in most if (possibly) not all language versions.

Other than that it is almost depressing how normal the sextortion campaign is: It uses the same spam sending infrastructure and the same low quality target address lists (the ones containing some subset of my spamtrap addresses) as the regular and likely not too successful spammers of every stripe. Nothing else stands out.

And as returning readers will notice, the logs indicate that the spambots are naive enough in their SMTP code that they frequently mistake spamd's delaying tactics for a slow, but functional open SMTP relay.

Now to recap the main points:
  • Regular users: The sextortion messages are scams, the videos do not exist. If this quasi-random sample is representative, the scammers are seen to send to 200 non-existing, invalid addresses before lucking on a real one. This alone strongly indicates that no videos exist. There is no reason to send money, bitcoin or otherwise. Look instead to learning how your devices and the networks and services they connect to actually work.
  • Competent mail admins: The tools to stop the flow of sextortion messages or at least slow to a manageable trickle are available today. You simply need to keep your antispam game up to speed with best practices and best of breed tools. If you are a user or someone who manages mail admins, check what your mail service does.
  • Competent authorities: Please step up to the task of educating the public. Sane, fact based approaches to IT security work. While it is easy to get distracted by the potential presence of porn and users' feelings of shame over accessing that kind of material, assigning much weight to that side of the matter is counterproductive. Work to educate the public and please focus on real threats, not imagined ones like the present topic.
Whatever evolves next out of these rather hamfisted attempts at blackmail is unlikely to ever achieve any level of sophistication worthy of the name.

We would all be much better served by focusing on real threats such as, but not limited to, credential harvesting via deceptive content delivered over advertising networks, which themselves are a major headache security- and privacy-wise, or even harvesting via phishing email.

Both of the latter have been known to lead to successful compromise with data exfiltration and identity theft as possible-to-probable results.

To a large extent the damage could could have been significantly limited had the general public been taught sensible security practices such as using multi-factor authentication or at least actually good passwords combined with securely coded password management applications, and insisting that services encourage such practices.

Yes, I know you have been dying to ask: What is the thing about Adnan? According to my activity log, the address adnan@bsdly.net was added as a spamtrap on July 8th, 2017 after somebot had tried to log on as the user adnan, a user name not seen before at bsdly.net,

Jul  8 09:40:34 skapet sshd[34794]: Failed password for invalid user adnan from 118.217.181.8 port 41091 ssh2

apparently from a network in South Korea.

As always, there is more log material available to competent practitioners and researchers with a valid research agenda. Please contact me if you are such a person who could use the collected data productively.


Update 2020-02-29: For completeness and because I felt that an unsophisticated attack like the present one deserves a thorough if unsophisticated analysis, I decided to take a look at the log data for the entire 7 day period, post-rotation.

So here comes some armchair analysis, using only the tools you will find in the base system of your OpenBSD machine or any other running a sensibly stocked unix-like operating systen. We start with finding the total number of delivery attempts logged where we have the body text 'am a hacker' (this would show up only after a sender has been blacklisted, so the gross number actual delivery attempts will likely be a tad higher), with the command

zgrep "am a hacker" /var/log/spamd.0.gz | awk '{print $6}' | wc -l

which tells us the number is 3372.

Next up we use a variation of the same command to extract the source IP addresses of the log entries that contain the string 'am a hacker', sort the result while also removing duplicates and store the end result in an environment variable called lastweek:

 export lastweek=`zgrep "am a hacker" /var/log/spamd.0.gz | awk '{print $6}' | tr -d ':' | sort -u `

With our list of IP addresses tucked away in the environment variable go on to: For each IP address in our lastweek set, extract all log entries and store the result (still in crude sort order by IP address), in the file 2020-02-29_i_am_hacker.raw.txt:

 for foo in $lastweek ; do zgrep $foo /var/log/spamd.0.gz | tee -a 2020-02-09_i_am_hacker.raw.txt ; done

For reference I kept the list of unique IP addresses (now totalling 231) around too.

Next, we are interested in extracting the target email addresses, so the command

grep "To:" 2020-02-29_i_am_hacker.raw.txt | awk '{print substr($0,index($0,$8))}' | sort -u

finds the lines in our original extract containing "To:", and gives us the list of target addresses the sources in our data set tried to deliver mail to.

The result is preserved as 2020-02-29_i_am_hacker.raw_targets.txt, a total of 236 addresses, mostly but not all in domains we actually host here. One surprise was that among the target addresses one actually invalid address turned up that was not at that time yet a spamtrap. See the end of the activity log for details (it also turned out to be the last SMTP entry in that log for 2020-02-29).

This little round of armchair analysis on the static data set confirms the conclusions from the original article: Apart from the possibly titillating aspects of the "adult" web site mentions and the attempt at playing on the target's potential shamefulness over specific actions, as spam campaigns go, this one is ordinary to the point of being a bit boring.

There may well be other actors preying on higher-value targets through their online clumsiness and known peculiarities of taste in an actually targeted fashion, but this is not it.

A final note on tools: In this article, like all previous entries, I have exclusively used the tools you will find in the OpenBSD (or other sensibly put together unixlike operating system) base system or at a stretch as an easily available package.

For the simpler, preliminary investigations and poking around like we have done here, the basic tools in the base system are fine. But if you will be performing log analysis at scale or with any regularity for purposes that influences your career path, I would encourage you to look into setting up a proper, purpose-built log analysis system.

Several good options, open source and otherwise, are available. I will not recommend or endorse any specific one, but when you find one that fits your needs and working style you will find that after the initial setup and learning period it will save you significant time.

As per my practice, only material directly relevant to the article itself has been published via the links. If you are a professional practitioner or researcher with who can state a valid reason to need access to unpublished material, please let me know and we will discuss your project.

Update 2020-03-02: I knew I had some early samples of messages that did make it to an inbox near me squirreled away somewhere, and after a bit of rummaging I found them, stored here (note the directory name, it seemed so obvious and transparent even back then). It appears that the oldest intact messages I have are from December 2018. I am sure earlier examples can be found if we look a littler harder.

Update 2020-03-17: A fresh example turned up this morning, addressed to (of all things) the postmaster account of one of our associated .no domains, written in Norwegian (and apparently generated with Microsoft Office software). The preserved message can be downloaded here. 

Update 2020-05-10: While rummaging about (aka 'researching') for something else I noticed that spamd logs were showing delivery attempts for messages with the subject "High level of danger. Your account was under attack."  So out of idle curiosity on an early Sunday afternoon, I did the following:

$ export muggles=`grep " High level of danger." /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u`
$ for foo in $muggles; do grep $foo /var/log/spamd >>20200510-muggles ; done


and the result is preserved for your entertainment and/or enlightenment here. Not much to see, really other than that they sent the message in two language varieties, and to a small subset of our imaginary friends.

Update 2020-08-13: Here is another snapshot of activity from August 12 and 13: this file preserves the activity of 19 different hosts, and as we can see that since they targeted our imaginary friends first, it is unlikely they reached any inboxes here. Some of these campaigns may have managed to reach users elsewhere, though

Update 2020-09-06: Occasionally these messages manage to hit a mailbox here. Apparently enough Norwegians fall for these scams that Norwegian language versions (not terribly well worded) get aimed at users here. This example, aimed at what has only ever been an email alias made it here, slipping through by a stroke of luck during a time that IP address was briefly not in the spamd-greytrap list here, as can be seen from this log excerpt. It is also worth noting that an identically phrased message was sent from another IP address to mailer-daemon@ for one of the domains we run here.

Update 2021-01-06: For some reason, a new variant turned up here today (with a second message a few minutes later and then a third), addressed to a generic contact address here. A very quick check of logs here only turned up only this indication of anything similar (based on a search for the variant spelling PRONOGRAPHIC), but feel free to check your own logs based on these samples if you like.

Update 2021-01-16: One more round, this for my Swedish alter ego. Apparently sent from a poorly secured Vietnamese system.

Update 2021-01-18: A Norwegian version has surfaced, attempted sent to approximately 115 addresses in .no domains we handle, fortunately the majority of the addresses targeted were in fact spamtraps, as this log extract shows.

Update 2021-03-03: After a few quiet weeks, another campaign started swelling our greytrapped hosts collection, as this hourly count of IP addresses in the traplist at dump to file time shows:

Tue Mar  2 21:10:01 CET 2021 : 2425
Tue Mar  2 22:10:01 CET 2021 : 4014
Tue Mar  2 23:10:01 CET 2021 : 4685
Wed Mar  3 00:10:01 CET 2021 : 4847
Wed Mar  3 01:10:01 CET 2021 : 5759
Wed Mar  3 02:10:01 CET 2021 : 6560
Wed Mar  3 03:10:01 CET 2021 : 6774
Wed Mar  3 04:10:01 CET 2021 : 7997
Wed Mar  3 05:10:01 CET 2021 : 8231
Wed Mar  3 06:10:01 CET 2021 : 8499
Wed Mar  3 07:10:01 CET 2021 : 9910
Wed Mar  3 08:10:01 CET 2021 : 10240
Wed Mar  3 09:10:01 CET 2021 : 11872
Wed Mar  3 10:10:01 CET 2021 : 12255
Wed Mar  3 11:10:01 CET 2021 : 13689 
Wed Mar  3 12:10:01 CET 2021 : 14181
Wed Mar  3 13:10:01 CET 2021 : 15259
Wed Mar  3 14:10:01 CET 2021 : 15881
Wed Mar  3 15:10:02 CET 2021 : 17061
Wed Mar  3 16:10:01 CET 2021 : 17625
Wed Mar  3 17:10:01 CET 2021 : 18758
Wed Mar  3 18:10:01 CET 2021 : 19170
Wed Mar  3 19:10:01 CET 2021 : 20028
Wed Mar  3 20:10:01 CET 2021 : 20578
Wed Mar  3 21:10:01 CET 2021 : 20997

and they attempted to get to mailer-daemon@, as can be seen from this preserved message as well as this one (both of which actually did inbox due to aliases).

Stay safe out there.

Update 2021-04-17: A new variant, somewhat crudely worded, inboxed today. Preserved here, here and here.

Update 2021-05-15: After swelling the list of trapped hosts considerably over the last few days, a sample of the campaign with the most correctly worded Norwegian text inboxed today, and I later found other samples.

From the logs it looks like the campaign started on May 13th:

 Thu May 13 10:10:01 CEST 2021 : 3998
Thu May 13 11:10:01 CEST 2021 : 4064
Thu May 13 12:10:01 CEST 2021 : 7052
Thu May 13 13:10:01 CEST 2021 : 7297
Thu May 13 14:10:01 CEST 2021 : 7474
Thu May 13 15:10:01 CEST 2021 : 10178
Thu May 13 16:10:01 CEST 2021 : 10251
Thu May 13 17:10:01 CEST 2021 : 11150
Thu May 13 18:10:01 CEST 2021 : 12686
Thu May 13 19:10:01 CEST 2021 : 12866
Thu May 13 20:10:01 CEST 2021 : 14708
Thu May 13 21:10:01 CEST 2021 : 14713
Thu May 13 22:10:01 CEST 2021 : 14907
Thu May 13 23:10:02 CEST 2021 : 16336
Fri May 14 00:10:01 CEST 2021 : 16360
Fri May 14 01:10:01 CEST 2021 : 16473
Fri May 14 02:10:01 CEST 2021 : 17608
Fri May 14 03:10:01 CEST 2021 : 17643
Fri May 14 04:10:01 CEST 2021 : 17671
Fri May 14 05:10:01 CEST 2021 : 17763
Fri May 14 06:10:01 CEST 2021 : 18796
Fri May 14 07:10:01 CEST 2021 : 18950
Fri May 14 08:10:02 CEST 2021 : 18972
Fri May 14 09:10:01 CEST 2021 : 18725
Fri May 14 10:10:01 CEST 2021 : 19929
Fri May 14 11:10:01 CEST 2021 : 19942
Fri May 14 12:10:01 CEST 2021 : 17046
Fri May 14 13:10:01 CEST 2021 : 18068
Fri May 14 14:10:01 CEST 2021 : 18619
Fri May 14 15:10:01 CEST 2021 : 16066
Fri May 14 16:10:01 CEST 2021 : 17468
Fri May 14 17:10:01 CEST 2021 : 17297
Fri May 14 18:10:01 CEST 2021 : 15859
Fri May 14 19:10:01 CEST 2021 : 17395
Fri May 14 20:10:01 CEST 2021 : 15934
Fri May 14 21:10:01 CEST 2021 : 15996
Fri May 14 22:10:01 CEST 2021 : 17120
Fri May 14 23:10:02 CEST 2021 : 16238
Sat May 15 00:10:01 CEST 2021 : 16299
Sat May 15 01:10:01 CEST 2021 : 16362
Sat May 15 02:10:01 CEST 2021 : 16346
Sat May 15 03:10:01 CEST 2021 : 16814
Sat May 15 04:10:01 CEST 2021 : 16812
Sat May 15 05:10:01 CEST 2021 : 16787
Sat May 15 06:10:01 CEST 2021 : 16007
Sat May 15 07:10:01 CEST 2021 : 17093
Sat May 15 08:10:01 CEST 2021 : 17101
Sat May 15 09:10:01 CEST 2021 : 17015
Sat May 15 10:10:01 CEST 2021 : 15702
Sat May 15 11:10:01 CEST 2021 : 15637

Update 2021-06-16: Another campaign seems to be under way, with this message sent to an address which I can reveal is simply an alias. 

Update 2021-08-16: Thanks to one particular operator being 'too big to block' this message, apparently part of a campaign that has at least 103 other sending hosts that are currently trapped here, actually inboxed despite being sent to a spamtrap which also corresponded to a forgotten alias for an actual in-use mailbox. 

Update 2021-08-17: By lunchtime the output of 

grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':' | wc -l

had reached 471, so I did 

export trash=`grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':'`
for foo in $trash ; do grep $foo /var/log/spamd >> vellykket.txt ; done

You can find the result here: vellykket_20210817T1200.txt. It looks like the campaign is still in progress.
 
A few hours later, the number was 570 and the new export looks like vellykket.txt while the most up to date list of IP addresses participating in the campaign is in vellykke_addressest.txt
 
If you're interested in further data, please let me know.
 
Update 2021-09-09: There are signs that another campaign is in progress, with an inboxed sample preserved here. This particular message appears to have been delivered from a Korean network.

Update 2021-10-27: Another sample inboxed overnight, from a campaign that uses a text with only slight edits from eariler.

Update 2021-11-29: Overnight a collection of trimmed-down messages like this one appeared, claiming to have installed a trojan on the supposed victim's phone, but asking the victim to answer the message for further instruction. An attempt to weed out spamtraps from their address lists, perhaps?

Update 2022-02-02: Another campaign is underway, a sample has been preserved here. It makes the usual claims of device takeover. This particular message seems to have been delivered via a Kenyan system.

Update 2022-03-30: A new entry appeared today, with only minor variations relative to earlier campaigns. As expected a log extract shows that the same host had been used in some spamming campaign or other -- possibly even an earlier segment of the same one -- only a few days ago.

Update 2022-04-08: The number of languages used in those messages received here grew by one with today's entry, which seems to be in German. I am not qualified to speak to the quality or lack of same of the translation, but I note that the host that was used to send the message seems to belong in an Indonesian network.

Update 2022-04-09: Yet another German language entry, this on also sent from a system apparently in Indonesia.

Update 2022-08-19: A new Norwegian language campaign is under way, with a handful of new samples available in the archive.

Update 2022-09-18: Another campaign in progress, this time picking up on quasi-recent buzzwords. I offer the evidence so far.

Update 2022-09-24: Yet another campaign, very similar to the last one. This message was apparently sent from a (likely compromised) Kuwaiti system.

Update 2022-10-06: Here we go again. The campaign has been going on for a little while, the first message to inbox (sort of) was this one, apparently delivered from a host located in Korea. The list of identified spam sources (246 hosts at this point) is here, while a log of activity can be found here. Warning: that last one is not a small file.

Update 2022-11-23: Another campaign is underway, with a variety of subjects, all with the word trouble, a full list preserved here. In addition, a few extracts, startin with the list of sending addresses (178 total) the list of target email adddresses (480 total) a full log extract covering the whole campaign while extract per host can be found in this directory.

None of this inboxed, of course, as they could all be found in the list of spamtraps, referenced among other places in my previous article The Things Spammers Believe - A Tale of 300,000 Imaginary Friends (also available without trackers.


If you have further data on these or similar incidents that you are able to share or if you want to look further into these and similar incidents, please let me know.

If you find any errors in the material I publish or disagree with my sentiments, or if you find this article interesting, useful or annoying, please let me know, either in comments or via email.


Saturday, December 28, 2019

The Year 2019 in Review: This Was, Once Again, Weirder Than the Last One

The year is 2019. By now Blade Runner is a movie about the past, but there are still bots out there trying to guess our passwords. It gets betterworse from here while the dictionaries expand.

The year is coming to an end and events during that year, as they happened, somehow lead to me leaving writing mainly to one side and blog posting only until I saw a bigger picture.

Now with only a couple of days left to go, we see that this year began much like the previous, with a not too bright set of bots endlessly trying to guess passwords. But on January 2, a new development caught my eye:


It was fairly obvious that some bot operator had the columns in their database mixed up, and I found the episode so laughable myself that I did not even bother to include it as the local part of a spamtrap. But as we will see later, it was an early sign of things to come. As you have probably suspected, the ssh password guessing activities have continued at pace, yielding this year so far

[Sat Dec 28 17:01:28] peter@skapet:~/website$ grep 2019 spamtraps-dateadded.txt | grep -c SSH
51233


that is, the local part of more than fifty thousand spamtraps.

In addition the early part of the year saw several campaigns of the email scams trying to extort various Bitcoin amounts in return for not publishing supposedly embarrassing videos, one of which I tweeted about:

You should be able to find further absurdities of a similar kind by looking for the hashtags #blooper_reel from that tweet as well ast #turbators. With those hashtags you will notice that there is at least anecdotal evidence that messages of the same kind have been directed at a significant subset of our spamtraps here (which for obvious reasons would not have been used in connection with any actual user login anywhere), evidenced by the spamd(8) log snippet preserved in this tweet:

And as noted in the followup tweet, other weirness was already happening:

More specifically, in the overnight haul on the morning of January 30th, I noticed via my scriptery that reports on such things that a large number of apparent bounce message deliveries to addresses made up of "Western-firstname.Chinese-lastname@mydomain.tld", such as aaron.pu@bsdly.net or abby.na@bsdly.net, had turned up, in addition to a few other varieties with no dot in the middle, possibly indicating separate sources.

That initial overnight batch only had only a couple of hundred new potential spamtraps in it (as evidenced by the spamtraps added log), but even at that point the greylist data seemed to indicate that the bounces were produced by a relatively small set of IP addresses in Chinese networks. We see such bursts at times, but they rarely last long, so at first I did not think much of it before simply adding those addresses as spamtraps.

This was one round that kind of exceeded expectations, in that what we can only conclude was the noise generated by one or more phishing campaigns targeting Chinese users lasted well into April of this year and ended up yielding more than 120,000 "imaginary friends", or spamtraps as others would say. It is likely that each of those fake addresses were used more than once, and in this context we only count new ones, so the actual number of messages and users targeted was probably a lot larger than the number of faked email addresses found in our logs here.

The delivery attempt from this tweet may well have been a product of the same campaigns:


By this time whoever was behind the campaign may have acieved their goals and moved on, or we could hope that they had been shut down by competent authorities.

But back to the password guessers, sometimes referred to as The Hail Mary Cloud. We have seen amazing feats of incompetence on their part before, but I seriously thought we had reached peak when some bot tried to log on a system in my care as the user "*" (yes, asterisk):


It should be noted of course that this confused my very much grep(1)-based script that among other things turns up new candiates for spamtraps. But again it was an early indication that by their incompetence at least some of the bot herders had exposed their methods. Weird things turn up on occasion, but it took until October before it dawned on me that at least some of the password guessing bots could be running with their username and passwords fields swithched around:


A few days later I stopped trying to write a witty article about the phenomenon:

but I kept harvesting new entries for the local parts of spamtraps, while noting that my still grep(1)-centric script for detecting candiates would relatively frequently fail while trying to interpret what looked like regular expressions, with messages such as

grep: repetition-operator operand invalid
-bash: [: ==: unary operator expected


turning up instead.

Some of these entries (this month's worth so far* can be found in this file) were weird enough (would you actually have created a user called !@#$%^&*()dianlut+_ ?) that they had me thinking that the operators of those bots were actually trying to be smart by working from stolen and published collections of password hashes.

The wrinkle to this could turn to our advantage is that some of these operators managed to get the order of their fields wrong and are throwing either raw password hashes or decoded ones at our systems instead of matching user names. By reversing the process we might be able to see which collections are used, or other weird and creative things.

If you are interested in doing further research on this, please contact me by email or the comments. I consider the traplist data, the dates added log and the other material mentioned in this piece and links therein to be public and the data should be available to anyone. Howerer, some data exists only in some more detailed logs that are preserved here only to be seen by competent eyes and used for valid purposes. If you consider yourself such a person (aka a professional), please feel free to contact me.

All the while, we see the dictionaries of user names and passwords expanding, and I for one am more than willing to help out in the effort. It helps us all identify the never-do-wells as early as possible in the game.

The raw numbers for our contributions to the hopefully confusing dictionary as they stand right now are (they will be different when you read this):

We have a total of 242778 spamtraps, with the numbers added according to the dates added log this year at 131195 from SMTP traffic, 51233 from failed SSH login attempts and 11 innovations from POP3 logon attempts.

This means our list of spamtraps did not reach a full quarter milllion this year.

But I already sense that somebody, somewhere is about to say "Hold my beer".



* Update 2020-01-02: This file now has the complete, or as complete as can be with the current scriptery, list of usernames tried during the whole month of December 2019.

Update 2020-01-07: You would probably not notice from looking at the raw listing of attempted usernames so far this month, but the theme so far in 2020 among the new arrivals seems to be, of all things, three letter user names (take a peek at 2020 part so far at the end of the spamtraps added log). Go figure.