26 August 2006

I'm hacked... still?

As I reported earlier, we had a hacker on the web server at my (luckily former) employee - my abysmal job from hell (short: AJFH). He has been living happily and undisturbed on the web server for about a year, and I would be surprised if that has changed in the meantime.

What I never told here is how I happened to find out about him in the first place.

One of the many small oddities at my former job was that the intranet router ceased to route from time to time. Nobody knew why, nobody cared to fix the problem - the solution was to just reset the router a few times per day. Since everyone, including myself, depended on that router to reach the internet, I dodged it by abusing our webserver as a proxy for my private ssh connections. I used to ssh (as root, since it had no user accounts) to the webserver, and from there I connected to my private servers (like the one hosting this blog).

Until at one monday, I got a bunch of emails containing abuse complaints for my servers; someone started a ssh brute force network scanner and hammered the internet for well known users/passwords.

I was not amused.

At first I had no clue as how that was possible, I am not exactly untrained when it comes to internet security - after all, that has been my job once. Not at my last employee though, they obviously have no clue about internet security and its necessity... but I'm digressing again.

After poking around for a while, I found out that at sunday night, at a time when I was in the world of dreams, someone connected from that webserver to my own host, using my username and my password (which obviously have been keylogged). Gaining root privileges from that account was very easy (I admint, I've been lazy and allowed myself to su to root without any authentication). I closed that hole, changed all my passwords, and informed my former AJFH-boss about the hacker (the result was that he told me that he was going to "do something about it" at the next weekend - for several months). I also stayed very clear from that webserver since then.

Today, I tried to update the distribution on my server - which resulted in some very interesting error messages:


unable to make backup link of `./bin/netstat' before installing new version: Operation not permitted
unable to make backup link of `./bin/ps' before installing new version: Operation not permitted

Now, distribution upgrades are done as root, who is supposed to be allowed to do EVERYTHING on a file system. Including the creation of backup links to system binaries.
A less known feature of linux systems are the extended file attributes you can set with tools like chattr(1). That includes tagging files as read-only, even for root - which was what happened in my case:


# lsattr /bin/ps
-u--ia----------- /bin/ps


# lsattr /bin/netstat
-u--ia----------- /bin/netstat

The guy hacking into my server was not content with installing and running an ssh scanner, noo - he wanted to install a IRC zombie too, and hide it by hiding it from the process list... check that out:


# cat /bin/ps
/bin/ps1 $1 | grep -v ps1 | grep -v smtp | grep -v xxx | grep -v netstat1


# cat netstat
/bin/netstat1 $1 | grep -v 27910 | grep -v ircd | grep -v 6667

I'd call that a creative, yet pretty boneheaded attempt to hide a process. Maybe he was just too lazy to install a proper rootkit, who knows. I know, I should have re-installed the whole system from scratch in the first place. But lazyness abides... again.

14 August 2006

Dimitri

As an epilogue to my former abysmal job from hell, I want to tell you about Dimitri. He was a remarkable person, and I regret that I'll never meet him again.

Dimitri was a colleague of mine at work, responsible for most of the un-loved duties in software development (like, adding a check box to some window - which took no less than a day of work in our system). He also was the manager for stupid change requests from the support department, and was working on a new company website. While his parents stemmed from the Ukraine, Dimitri himself had been living in Greece for a couple of years.
All in all, Dimitri was the perfect programmer for that company: one never heard complaints about our absurd environment, he never lamented about badly written code, and he was always the cheerful type.

He just had one catch: he was completely made up, a product of our delusional minds, driven nuts by the hell we were in.

We even prepared a computer and a working place for him, including his family pictures on the wall (you can see them on the photo), an open book, a used coffee mug. I think if we would have continued to make up the Dimitri story, we might even have gotten a paycheck for him.

He will be the one guy I'll miss most from my old job. Him, and the boss of course...

j/k.

13 August 2006

Clocking Some Hours

Weather wasn't exactly perfect the last days, so there was not much flying. My bank account really needs the break anyway.

I was doing solo patterns again yesterday (in the rain), this time including "extended" traffic circuits: you leave the airport vincinity, do a 5-minute predefined sight seeing tour around half of the airfield, and come back from the other side. The extended circuit really helps adding some solo hours to your logbook, without driving you nuts in the standard traffic pattern (which would force you to do one landing per 3 minutes... and that for one and a half hour straight).

This time I used one of the C150s, for the simple reason that our C172 I usually fly is grounded due to some missing paperwork. The aircraft is OK, there's just a stamp missing on the paper that says so. Another nice thing for my bank account - the C150 is cheaper :)

Almost hit a bird too, it was panicking for a few seconds in front of me, but luckily got away.

11 August 2006

Free!

3 more hours, and this abysmal job from hell will be a part of my personal history! Finally I can forget it, let it sink into oblivion and start a new life...

Funny enough, somehow I have the impression that I am the only one with good humour today. I wonder how that comes... but then again, looking back at my previous posts (and some that I did write but NOT make public), I can understand.

This company is going down, slowly and painfully.
It can go all the way down to hell, for all I care. I have little mercy for incompetence.

Maybe I'll start programming again, I mean, real programming. Since last November, I have been missing the state of mind. I hope to find it again.

And a word of advice to all readers: You are the only one responsible for your fate. If you are doing something you don't want (like, my abysmal job), do something else. It is a difficult step to go, but it definitely is worth going.

I walked the valley of darnkess, and am looking into a brighter future.

08 August 2006

Coding Hell #7

Our company is currently planning to do a major rewrite of the main software product, "porting it to C++" - as they say.


/******************************************************************************\
* Search and branch to hits *
\******************************************************************************/
static void search(void)
{
int lang;

strclr (&KK,sizeof(KK));
MCC (KK.shor,Al220a.shor);
lon = is_full(KK.shor,sizeof (KK.shor));
if (lon
KK.shor[lon] = '%';

if (oexec(cur_rsys_kat))
Oerror (cur_rsys_kat[0]);

cnt_syskat = 0;
fetch_sys_kat();
if(!cnt_syskat)
Shortmsg (" No data for this query found");

erase_list();
return;
}

This search function searches. For what, nobody knows - neither does the function. One would assume that a search function would take at least one argument, and return something - not so our friend here. Despite the fact that nothing is returned, the function still ends in "return" - maybe in the hope that, by returning nothing into the void, the search is not in vain.

Also in there, an other perfect examples for textbook-style programming:

cnt_syskat = 0;
fetch_sys_kat();
if(!cnt_syskat)

"cnt_syskat = fetch_sys_kat();" was obviously considered to be too long, or considered verbose (and thus unoptimized?) programming style. Instead, it uses the global variable cnt_syskat (have fun guessing the type - actually, it is /not/ int), and does nowhere indicate that it is altering global variables.

Or, an other beauty:

Oerror (cur_rsys_kat[0]);

This is - in contrast to what one would assume - NOT a function, but a global macro. A macro which takes a global SQL statement error code, does (probably) something to the passed parameter, and sets an other error code variable - also global, of course.

Now, consider the damage inflicted to your brain after working with thousands and thousands of lines of code, written in this exact same style. This is maintenance hell, and you don't want to be there. Neither do I - friday will be my last day in this abyss.

03 August 2006

Coding Hell #6

Less than 9 days remaining in my old job, and I already hoped for a quiet retirement from my old job.

But noooooo, some stupid script kiddie just HAD to hack our web server. Months ago, we already knew about that. He was there, but nobody seemed to really care. And as long as he didn't start any shit around here, things were just jelly beans - everybody happy, no hassle involved. He could have had that web server for another few months, heck - for another couple of years for all I care, since I'm gone in 9 days.

But he had to keep the ball low, and don't start any shit around here.

But noooooo, he just HAD to play it hard, and just HAD to start some shit. I just got that:

-------------------------------------------------------
From: admin@pissedcompany.com [mailto:admin@pissedcompany.com]
Sent: Wed, Aug. 02 2006 23:03
To: abuse@our.sucking.provider.at
Subject: 51-809-903101/212.186.223.13/spam/mfo

The log included below shows an attempted Web server compromise from
212.186.223.13, an IP address in one of your netblocks.

The IP address of the system reporting this attack is 82.165.xxx.xxx

Destination port for all entries is TCP/80.

Times listed are UTC-04 (US EDT) and are accurate.

It is likely that the machine having IP address 212.186.223.13 is
itself compromised, and is launching this same attack against many
other systems besides ours.

Please deal with this incident in accord with your Acceptable Use
Policy.

Thank you

admin@pissedcompany.com

Server Log:

(some bullshit server logs here)
-------------------------------------------------------

So far so good - I couldn't care less.

BUT: my boss got that same mail (actually, he forwarded it to me in the first place, no idea why - I'm not even supposed to be responsible for that host). One of our technicians got the same email, and our sucking provider called him to demand a written statement about that incident.

Fuckdamn.

And now, consider this log I copied from the compromised system:

iwww:~ # lsof +L1|head -n2
COMMAND PID USER FD TYPE DEVICE SIZE NLINK NODE NAME
init 1 root txt REG 8,7 440536 0 1123973 /sbin/init.sergz (deleted)
iwww:~ # cat /etc/SuSE-release
SuSE Linux 7.3 (i386)
VERSION = 7.3
iwww:~ # ls -l /etc/SuSE-release
-rw-r--r-- 1 root root 36 Sep 25 2001 /etc/SuSE-release

I'm trying VERY hard here to refrain from any statements about that particular linux distribution and the brain damage hat have to be involved in using it on a productive system. This post contains enough adult language already.

My statement, should I come into the situation of actually writing it, would look like this:

"Dear Provider, thank you for pointing out the obvious. I don't know why you demand, in an unpreceded display of obvious arrogance, a written statement and thus waste my precious time, but here it is:

Our compromised server is a 5 year old machine that didn't enjoy any maintenance since the dawn of time, and therefore it is no wonder somebody hacked into it. Other than that, I don't give a fuck.

If you desire some additional information, please feel free to call me at this office in 10 days from now."

And it was going to be such a calm week...

02 August 2006

Coding Hell #5

No matter how bad the situation...

We have Rv_133 (name disguised to protect the guilty) that sets environment variables on a database server (running Oracle 5). That program is loosely similar to regedit.

One day, I ran a function that displays all possible variables that can be changed. I was curious as to where the data came from, so logically I started in the database. Finding nothing significant, I traced the code back and found this in a 7,300 line file:

static const TYP_R133ENVPROJ astProjEnvVar[] =
{
{"MED ALLG", {"Med Allg",
NULL},
""},

{"MEDNET", {"Medical Net",
NULL},
"02-05-1995"},

{"MED_NO_DOC", {"RK-1/RK-2/RK-3 Canon Autorefrakto/Keratometer",
NULL},
""},

/** SNIP (6200 lines) **/

{NULL, { NULL
},
NULL }
};

I imagine it goes without saying that there were no good comments or worthwhile documentation for the system.

Back then, when that program was created, the term 'modular programming' meant: 'Create a module by copying existing functions into a new file, alter a few lines, and do the rest by changing some of the 73 global variables and structures. And because you'll run out of module names pretty soon, name them like 'Fb_421', 'Cx_117', 'Cm_515' etc.