Summary: A cracker caused software to run at bootup, but the administrator couldn't figure out how.

(Note: The contest is, unfortunately, over. However the answer is available already if you're interested.)

Every now and then I like to issue challenges to my readers. This week you have another chance to test your wits against a problem, but the rewards are higher. Instead of a measly postcard, the winner will get a copy of Hacking Linux Exposed, Second Edition. I'll pick the winner from those who figure out the problem, based on the quality of the analysis of the problem described below. Anyway, onto the problem.

A user wrote me in a exasperated state. His machine had been cracked (his root password was less than stellar) and the cracker had installed various user-level rootkits and software. He had done a good job of removing the rootkits and cracking tools. When booting off CD (thus with no chance for any kernel modifications) he could see that there was nothing left that hadn't been accounted for. All the /etc/rc?.d directories were as they should be, no suspicious entries in /etc/xinetd.conf or /etc/xinetd.d.

He had no file integrity tools on this system (Tripwire, AIDE, etc) but he did verify each and every file that had a modification date and/or change date from the time the cracker entered, and nothing suspicious was in them.

The problem was that every time he booted the machine itself, several bizarre processes were started:

  # ps -ef | grep qw
  root     15    1      0 11:24 ?        00:00:00 /tmp/.qw/moniker
  root     242   15     0 11:25 ?        00:00:00 /tmp/.qw/cr8
  root     244   1      0 11:25 ?        00:00:00 /tmp/.qw/rucc

He checked the /etc/rc?.d directories numerous times and was absolutely sure nothing therein would start these processes, or start processes that would start those processes, and so on. He also checked cronjobs, atjobs, both those in the /var/spool/cron directory and the /etc/cron.* directories, and didn't find any that were related to this problem.

If he booted the machine, there was no /tmp directory (it was created on bootup every time from scratch) and if he deleted /tmp/.qw just to be safe, it was re-created somehow at the next bootup anyway. Some of these processes seemed to watch the local system, while the rucc created a network rootshell on port 7759.

To be absolutely paranoid, he deleted all rc?.d files and booted into single user mode, and the processes still somehow appeared from nowhere.[1]

If the machine were booted off clean media, these processes didn't start.

He re-installed all his packages (by booting off CD, so any local system changes by the cracker couldn't have any effect) but still no luck, these programs came out of nowhere. He searched the rest of the computer for any files with similar names, but no luck.

Somewhere, somehow, these files were appearing and being executed any time the system was booted off the hard drive. Your mission: figure out where he was forgetting to look, and how the cracker managed to re-install his software each time. The answer is simpler than you might expect, and didn't involve any kernel changes (recompilation or LKMS to start the software or hide how it was done) or trojaned software, so don't waste your energy on that line of reasoning.

Send your guesses to me at bri@hackinglinuxexposed.com. I'll pick one of the best entries and a copy of Hacking Linux Exposed, Second Edition will be on it's way to you as soon as it's printed.[2]

NOTES

[1] This was a Debian system, which has an /etc/rcS.d directory for single user mode and bootup scripts, but he cleaned that out too.

[2] Should be any day now. I really need to update our website...

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He loves to investigate compromised machines. He'd love it even more if machines were compromised less. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002

This is the November 27, 2002 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.