issues with security
bradbartram at ccsisp.com
Mon Feb 2 13:45:48 EST 2004
Getting hacked tends to be a major pain in the butt, but is one of those
things that falls into the category of "whatever doesn't kill us makes us
With specific regard to your problem here is my opinion...
You did not provide any specific system details regarding the running system
that got hacked except for the root kit used. When doing a forensic analysis
on a hacked system, always try to piece things together. Check your logs,
check the passwd and shadow file to see if any new entries have been added.
Some nice commands are:
netstat -planet - this one will show all tcp ports and the backround process
their bound to.
ls -halt - shows all files in a directory sorted by time
Aside from that, without proper infrastructure setup like external intrusion
detection and log servers, any data hosted on teh compromised server is
suspect and should be treated as tainted.
Based upon your description of steps taken to restore I would look in two
1 - your php code. There are many vulnerabilites relating to php that come
as a direct result of bad programming practices. Global variables, ftp
uploads, improper user input validation, etc, are all things that can
contribute to a security breach. If you have a bad php script that allows
for some user to execute arbitrary code as the apache user, it is relatively
trivial to elevate priviledges into a root compromise, at which time your box
is now the attackers.
2 - If you are sure your php code is good, I would look internally. Most
security compromises are internal in origin. I don't know the size of your
organization but I would do a thorough audit on the employees. It's a sucky
job to have to do, but that is the price of vigilance.
As a remediation, I would get any relevent data from teh system or even
install a new drive into the box (thus leaving the compromised drive in a
pristine condition for later use) and then reinstall your os. Apply all
errata and reinstall your app server. INSTALL AN MD5 BASED CHECKSUM
VALIDATION SOFTWARE - like tripwire - keep the signatures safe on a read only
medium like cd-r. Restore from a known good backup. If the one you used is
suspect - don't use it, go older. Do a line by line audit on the code to
make sure there are no trojans lurking on there. Enjoy your new server.
To round out the environment, I would install an external IDS like snort as
well as an external syslog server to monitor any anomolies in comfort. It
seems like someone has your number so you have to be extra cautious.
If you need any more advise or on site assistance, feel free to contact me off
On Monday 02 February 2004 12:45 pm, cliff at cliffmeyers.com wrote:
> Hi Everyone,
> Apparently my first message didn't go through, so here I go again:
> I've been away from the list for a little while, but been having a major
> problem here at the office so I figured I'd post to see if you guys had any
> On the 22nd we had an issue with one of our systems that I thought had to
> do with some kind of hard drive error. The system is a Red Hat Linux box,
> running primarily Apache and PHP to serve web sites. I typically compile
> these things from source so I can have a little more control over
> Anyways, as it turned out, I noticed late last week that there were
> processes running that shouldn't be there. After I killed the processes I
> noticed files in the /tmp directory, where PHP stores most of the session
> files (unless I tell it to store them somewhere else). There was a
> 'blackhole.c' file and some other things which had been compiled to run on
> my system.
> I talked to my other programmer and we were going to come in Saturday to do
> a full re-install, but the hacker struck against Thursday night around 11
> PM and defaced all of our sites. I came into the office and spent the next
> 8 hours formatting, installing Red Hat 9, download all the newest source
> code for Apache and PHP, and getting everything set up.
> Well, I get into work today, and guess what? Another bad process and more
> files in the /tmp folder. I killed them all again, and am going to do
> *another* reinstall tonight. I was e-mailing a colleague asking for his
> input so I'll post a few of the ideas I had for how the hacker got back in.
> Here they are:
> (1) I used the latest stable version of PHP, 4.3.4, when in fact there is
> a new version, 4.3.5RC1. I wanted to avoid a release candidate version but
> that would be my first guess.
> (2) Some other vulnerability I don't know about. I installed the latest
> version of every other package so that's probably unlikely. Every other
> service is firewalled, so...
> (3) I used the web backup from the morning of the 30th so as to not loose
> any changes - perhaps there was something in there that it allowing the
> hacker to gain access again.
> (4) A problem with some of our PHP code. Again, not sure how that's
> possible or what the issue might be.
> Does anyone have any other ideas? Can anyone direct me (or offer) security
> consulting services to help take a look? Is there any other information I
> can provide? This is the first time I've really dealt with this and my
> blood pressure is through the roof... thanks very much guys.
> -Cliff Meyers
More information about the nflug