The Year 2019 in Review: This Was, Once Again, Weirder Than the Last One

The year is 2019. By now Blade Runner is a movie about the past, but there are still bots out there trying to guess our passwords. It gets betterworse from here while the dictionaries expand.

The year is coming to an end and events during that year, as they happened, somehow lead to me leaving writing mainly to one side and blog posting only until I saw a bigger picture.

Note: This piece is also available with trackers but nicer formatting here.

Now with only a couple of days left to go, we see that this year began much like the previous, with a not too bright set of bots endlessly trying to guess passwords. But on January 2, a new development caught my eye:

The the present round of #hailmary #ssh #password gropers are not run by are the sharpest knives in the drawer, it seems:

Jan 2 21:29:48 skapet sshd[8180]: Failed password for invalid user password321 from 5.196.69.109 port 57634 ssh2

And more where they came from of course.
— Peter N. M. Hansteen (@pitrh) January 2, 2019

It was fairly obvious that some bot operator had the columns in their database mixed up, and I found the episode so laughable myself that I did not even bother to include it as the local part of a spamtrap. But as we will see later, it was an early sign of things to come. As you have probably suspected, the ssh password guessing activities have continued at pace, yielding this year so far

[Sat Dec 28 17:01:28] peter@skapet:~/website$ grep 2019 spamtraps-dateadded.txt | grep -c SSH
51233

that is, the local part of more than fifty thousand spamtraps.

In addition the early part of the year saw several campaigns of the email scams trying to extort various Bitcoin amounts in return for not publishing supposedly embarrassing videos, one of which I tweeted about:

Once more for the #blooper_reel, this time in German, TXT w/headers https://t.co/xcH0vmBQYs, PNG https://t.co/DnvMZrbCAG - note that all those messages come with List-Unsubscribe: headers. How utterly nice of them. pic.twitter.com/OhYWvsEtVX
— Peter N. M. Hansteen (@pitrh) January 11, 2019

You should be able to find further absurdities of a similar kind by looking for the hashtags #blooper_reel from that tweet as well ast #turbators. With those hashtags you will notice that there is at least anecdotal evidence that messages of the same kind have been directed at a significant subset of our spamtraps here (which for obvious reasons would not have been used in connection with any actual user login anywhere), evidenced by the spamd(8) log snippet preserved in this tweet:

I almost wish they'd sent to actual users' addresses so I could see the full text:

Feb 11 11:08:17 skapet spamd[79149]: 74.6.131.123: Body: s letter is not a hoax mail and I urge you to treat it serious. This letter=

(another wankvid-hoax-blackmail attempt most likely).
— Peter N. M. Hansteen (@pitrh) February 11, 2019

And as noted in the followup tweet, other weirness was already happening:

Also in a parallel development, the list of imaginary friends at https://t.co/3uthWgKWmL rolled past 64K today (a pointless statistic to anyone except me, I know), proving that if you do something for long enough, true absurdity is achievable.
— Peter N. M. Hansteen (@pitrh) February 11, 2019

More specifically, in the overnight haul on the morning of January 30th, I noticed via my scriptery that reports on such things that a large number of apparent bounce message deliveries to addresses made up of "Western-firstname.Chinese-lastname@mydomain.tld", such as aaron.pu@bsdly.net or abby.na@bsdly.net, had turned up, in addition to a few other varieties with no dot in the middle, possibly indicating separate sources.

That initial overnight batch only had only a couple of hundred new potential spamtraps in it (as evidenced by the spamtraps added log), but even at that point the greylist data seemed to indicate that the bounces were produced by a relatively small set of IP addresses in Chinese networks. We see such bursts at times, but they rarely last long, so at first I did not think much of it before simply adding those addresses as spamtraps.

This was one round that kind of exceeded expectations, in that what we can only conclude was the noise generated by one or more phishing campaigns targeting Chinese users lasted well into April of this year and ended up yielding more than 120,000 "imaginary friends", or spamtraps as others would say. It is likely that each of those fake addresses were used more than once, and in this context we only count new ones, so the actual number of messages and users targeted was probably a lot larger than the number of faked email addresses found in our logs here.

The delivery attempt from this tweet may well have been a product of the same campaigns:

Possibly in the spirit of the upcoming holidays, this morning somebot in .cn tried delivering mail to new imaginary friend Jesus Mao:
Apr 17 06:08:51 skapet spamd[32133]: new entry 106.13.14.227 from <ywlxo@funnyconsult.com> to <jesusmao@bsdly.net>, helo https://t.co/WC2xsKJs5N
— Peter N. M. Hansteen (@pitrh) April 17, 2019

By this time whoever was behind the campaign may have acieved their goals and moved on, or we could hope that they had been shut down by competent authorities.

But back to the password guessers, sometimes referred to as The Hail Mary Cloud (original). We have seen amazing feats of incompetence on their part before, but I seriously thought we had reached peak when some bot tried to log on a system in my care as the user "*" (yes, asterisk):

This must mean we have reached peak something:

May 31 05:48:50 skapet sshd[35089]: Failed password for invalid user * from 178.62.90.135 port 53721 ssh2
— Peter N. M. Hansteen (@pitrh) May 31, 2019

It should be noted of course that this confused my very much grep(1)-based script that among other things turns up new candiates for spamtraps. But again it was an early indication that by their incompetence at least some of the bot herders had exposed their methods. Weird things turn up on occasion, but it took until October before it dawned on me that at least some of the password guessing bots could be running with their username and passwords fields swithched around:

The overnight haul of new user names attempted for ssh logins looks a lot like somebot switched the usenames and password columns around. Probably what passes for innovation in those parts of the world.
— Peter N. M. Hansteen (@pitrh) October 4, 2019

A few days later I stopped trying to write a witty article about the phenomenon:

I have tried but probably will keep failing at writing a coherent article about the bot that apparently has its username and passwords fields reversed. To wit,
Oct 25 12:26:11 skapet sshd[78767]: Failed password for invalid user /']\\\\\\\\ from 188.254.0.226 port 51958 ssh2
— Peter N. M. Hansteen (@pitrh) October 26, 2019

but I kept harvesting new entries for the local parts of spamtraps, while noting that my still grep(1)-centric script for detecting candiates would relatively frequently fail while trying to interpret what looked like regular expressions, with messages such as

grep: repetition-operator operand invalid
-bash: [: ==: unary operator expected

turning up instead.

Some of these entries (this month's worth so far* can be found in this file) were weird enough (would you actually have created a user called !@#$%^&*()dianlut+_ ?) that they had me thinking that the operators of those bots were actually trying to be smart by working from stolen and published collections of password hashes.

The wrinkle to this could turn to our advantage is that some of these operators managed to get the order of their fields wrong and are throwing either raw password hashes or decoded ones at our systems instead of matching user names. By reversing the process we might be able to see which collections are used, or other weird and creative things.

If you are interested in doing further research on this, please contact me by email or the comments. I consider the traplist data, the dates added log and the other material mentioned in this piece (original) and links therein to be public and the data should be available to anyone. Howerer, some data exists only in some more detailed logs that are preserved here only to be seen by competent eyes and used for valid purposes. If you consider yourself such a person (aka a professional), please feel free to contact me.

All the while, we see the dictionaries of user names and passwords expanding, and I for one am more than willing to help out in the effort. It helps us all identify the never-do-wells as early as possible in the game.

The raw numbers for our contributions to the hopefully confusing dictionary as they stand right now are (they will be different when you read this):

We have a total of 242778 spamtraps, with the numbers added according to the dates added log this year at 131195 from SMTP traffic, 51233 from failed SSH login attempts and 11 innovations from POP3 logon attempts.

This means our list of spamtraps did not reach a full quarter milllion this year.

But I already sense that somebody, somewhere is about to say "Hold my beer".

* Update 2020-01-02: This file now has the complete, or as complete as can be with the current scriptery, list of usernames tried during the whole month of December 2019.

Update 2020-01-07: You would probably not notice from looking at the raw listing of attempted usernames so far this month, but the theme so far in 2020 among the new arrivals seems to be, of all things, three letter user names (take a peek at 2020 part so far at the end of the spamtraps added log). Go figure.