security (continued)

[cliff at cliffmeyers.com]

Mark,


First, let me thank you for all your comments and input.

I really highly doubt that anyone has hacked the firewall.  I suppose it's
possible, but considering it's a Sonicwall hardware firewall appliance I have a
hard time believe that anyone is able to just punch through that.  I'd have to
wager a guess that on average it's "more secure" than a Linux firewall box, but
then again, I could be totally wrong.

I've examined the other 3 servers in the DMZ - 2 are web running ColdFusion and
the 3rd is a database server running MySQL and PostgreSQL.  Naturally, the hacker
could have covered their tracks here, but considering the exploits on the PHP web
server were so recognizable I'd doubt that these would have been hacked in such a
secretive and well-disguised manner.

Anyways, I'm going to push onward with the re-install tonight.  Here's my guess
what happened - maybe you can tell me if you think I'm crazy or not:

(1)  Hacker runs vulnerability script against PHP-based server
(2)  Hackers finds it running PHP 4.3.2 and Apache 1.3.26
(3)  Hacker uses well-known exploits to gain root access
(4)  Cliff notices this and re-installs the OS
(5)  Cliff makes stupid mistake of restoring the web root of all web servers onto
new server and possibly transfers the exploit files back onto server
(6)  Hacker finds exploits and uses them again to get weird process running on
server (as of Monday morning)

What do you think?  Obviously there are an infinite number of paths that could be
taken here.  The other (possible, maybe plausible) explanation is a problem with
the PHP code on the server itself.  Either way I'd lean towards the problem being
related to PHP since the other "unaffected" servers aren't running PHP and 2 of
them are running Apache (including one with an older version on it) and haven't
been hacked...

Now I'm the one babbling!  I'll let you know how things go.


-Cliff



On Mon, 2 Feb 2004 15:14:07 -0500, "Mark Musone" wrote:

> 
> Since you gave this info, I also have to comment :)
> 
> It looks like your machine got totally hosed..once your dealing with
> _any_ root based program being installed/run on your machine, then most
> likely every system admin binary got replaced with a Trojan. That’s
> usually the second step a hacker takes to hide themselves.
> 
> I'd also look into either your firewall being compromised or a different
> machine behind your firewall being compromised..
> The me a very big red question should be "how is this person able to
> connect in on a port that’s blocked by the firewall" and it's either
> that the firewall got compromised, or a different machine behind the
> firewall (and therefore you web server is not firewalled against) got
> compromised..
> 
> -Mark
> 
> 
> -----Original Message-----
> From: owner-nflug at nflug.org [mailto:owner-nflug at nflug.org] On Behalf Of
> cliff at cliffmeyers.com
> Sent: Monday, February 02, 2004 2:45 PM
> To: nflug at nflug.org
> Subject: security (continued)
> 
> Everyone,
> 
> 
> Well, I found a few more things this afternoon.  I made a backup up the
> /etc,
> /tmp, /var and /root dirs before formatting the drive this past Friday
> night.  I
> found this in my /var/logs/messages.1 file:
> 
> Jan 21 16:42:43 titanium modprobe: modprobe: Can't locate module ppp0
> Jan 21 16:42:43 titanium kernel: k uses obsolete (PF_INET,SOCK_PACKET)
> Jan 21 16:42:44 titanium kernel: eth0: Promiscuous mode enabled.
> Jan 21 16:42:44 titanium kernel: device eth0 entered promiscuous mode
> Jan 21 16:47:52 titanium modprobe: modprobe: Can't locate module ppp0
> 
> This was the day before the night I noticed problems with our server -
> the
> cronjobs had frozen up, there were loads of bizarre processes running
> and when I
> tried to reboot the server I got a "segmentation fault" error on the
> first couple
> lines of bootup.  I upgraded to RedHat 9 and that allowed the server t o
> boot
> fine, but after the server was defaced this past Thursday night I wiped
> te hard
> drive clean.  The binary "k" in fact is the name of the program that
> showed up
> that night and ALSO this morning on my server's process list.  And I see
> they're
> turned my eth0 to promiscious mode so it can probably try and capture
> traffic on
> my network.
> 
> It looks like this "k" program is a backdoor that the hacker uses to do
> nasty
> things to my system, but in and of its doesn't do anything nasty.  I
> need to
> check my current /var/log/messages file to look for these same error
> messages.  I
> also need to search my server to see if I can find the file itself
> again.
> 
> I also was looking at the backup I made this past Friday and found two
> files in
> my /tmp directory, dc (binary) and dc_connectback.c.  Looks like it
> might be a
> different kind of backdoor used.  If anyone wants to look at the C code
> let me
> know and I can e-mail the file.
> 
> I still plan on formatting the disk again tonight, running up2date and
> doing a
> re-install with the latest version of PHP.  Once again, I compile the
> web-related
> stuff from source so I always use the latest tarballs when doing this.
> Then I
> need to slowly migrate my PHP code site-by-site and have my other
> programmer
> examine for backdoors or bad files.  If the server is compromised yet
> again after
> this, it's probably safe to say it's a problem with our PHP code, huh?
> 
> Oh, thanks for the tip about the internal problem.  I have one other
> programmer
> on my staff and no one else in the office really knows what Linux is!
> So I think
> I'm good there, but still a good suggestion.
> 
> Once again, any other ideas and comments are more than welcome.  I may
> very well
> take some of you up on your offers to come in and look at my server and
> help me
> harden it.  Thanks again, brain trust!
> 
> 
> -Cliff
>