Our Continuing /proc and lsof Investigation
By Bri Hatch.
Summary: Following up on last week's useful /proc and lsof investigative tools, Bri goes back and addresses the tidbits he didn't have time for last week.
Let's take a quick look at the files available for our running mystery process (which we've already determined is likely the John the Ripper password cracker) to see what we can learn. The process id was 24061, so let's look at the files therein:
$ cd /proc/24061 $ ls -r--r--r-- 1 axe axe 0 May 13 13:42 cmdline lrwx------ 1 axe axe 0 May 13 13:42 cwd -> /tmp/r (deleted) -r-------- 1 axe axe 0 May 13 13:42 environ lrwx------ 1 axe axe 0 May 13 13:42 exe -> /tmp/r/john (deleted) dr-x------ 2 axe axe 0 May 13 13:42 fd pr--r--r-- 1 axe axe 0 May 13 13:42 maps -rw------- 1 axe axe 0 May 13 13:42 mem lrwx------ 1 axe axe 0 May 13 13:42 root -> / -r--r--r-- 1 axe axe 0 May 13 13:42 stat -r--r--r-- 1 axe axe 0 May 13 13:42 statm -r--r--r-- 1 axe axe 0 May 13 13:42 status
Some of these bits we've already seen interpreted by ps or lsof, such as the cwd (current working directory) and the files inside fd (the open file descriptors.) The cmdline file contains the command line that was used to start the program -- or whatever the program altered it to. These come straight out of the argv array, for you C programmers. Each argument is separated by a null, which means that it looks all jumbled together if you simply 'cat' it. Say I launch 'find / -name \*.mp3 -exec chmod a+r {} \;' in a window and look at it's cmdline entry
$ cat cmdline
find/-name*.mp3-execchmoda+r{};$
That '$' at the end is our prompt. Pretty damned unreadable. An easy way to parse out the command line usably would be to use this short command line perl script:
$ perl -pe 's/\0|$/\n/g' cmdline
find
/
-name
*.mp3
-exec
chmod
a+r
{}
;
The environ file has a complete listing of all environment variables that are set by the program in question. You could get this list using 'ps eww' for example:
$ ps eww -p 24061 PID TTY STAT TIME COMMAND 24061 tty3 R 4:52 ./john smarmy _=./john PATH=/usr/local/ bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/bin:/opt/sbin:/sbin:/usr/ sbin:/etc:/usr/etc:/var/yp:/usr/lib:/usr/local/sbin:/usr/games: /home/bri/bin:. SHELL=/bin/ksh EDITOR=vi HOSTNAM....
But that's pretty darn ugly. Instead you could parse the environ file (it also uses nulls between entries) just like we did the cmdline file:
$ perl -pe 's/\0|$/\n/g' environ | sort | head -6 DISPLAY=localhost:10.0 EDITOR=emacs HOME=/home/axe HOSTNAME=rooted HOSTTYPE=i386 LANG=en_US
It would seem that the attacker didn't cleanse his environment at all, and we may have some useful info in here.
The maps file shows memory mappings in use by the program:
$ head -2 maps 40000000-40013000 r-xp 00000000 03:07 40255 /lib/ld-2.1.3.so 40013000-40014000 rw-p 00012000 03:07 40255 /lib/ld-2.1.3.so
These were already visible in lsof output, but you get more information about them through the maps file, if you are savvy enough to use it.
The mem file is an interface directly into the process' memory, meaning you could read (or change) the real bytes that make up the program and it's data. Stat, statm, and status all contain (sometimes repetitive) status information about the process. See the proc(5) man page for a detailed description. The status file is the most humanly readable version:
$ head -6 status Name: john Stat: R (running) Pid: 24061 PPid: 1087 Uid: 1020 1020 1020 1020 Gid: 1100 1100 1100 1100
Note that the 'name' here is the original program name, and cannot be modified by changing argv. This is the name that lsof uses.
The two files cwd and root are symbolic links that point to the process' current working directory and root, respectively. Using the chroot system call, you can restrict a program to live entirely within a specific directory, unable to access files outside of the chroot area. This is common with security-conscious network services. In our case the process is running from a deleted /tmp/r directory, with no special chrooting.
That pretty much covers the important uses of /proc/PID meta-files. However we're still stuck with a question about our mystery password cracking process: since it's running with all the files it uses already deleted, how will the attacker be able to recover these files when the password cracker is finished? It may be that he's modified John to copy the decrypted passwords somewhere else when done, or send them to him in email, or other possibilities. Since the cracker is only running the one process, this seems the most likely answer. However it's worth checking if any other processes open those deleted file descriptors:
$ lsof | grep deleted save 24070 samson 4r REG 3,5 39 49446 /tmp/r/john.pot (deleted) save 24070 samson 5r REG 3,5 57 49447 /tmp/r/restore (deleted) save 24070 samson 6r REG 3,5 16157 49435 /tmp/r/password.lst (deleted)
Hmmn...seems the user samson has a program called 'save' with access to those same deleted files. Even when John quits, those files will be accessible to the cracker to copy or send as desired. Perhaps the cracker thought that using a different user would fool us. Or perhaps samson is the real mischievous entity, and is trying to get 'axe' in trouble, but reap the rewards of the password cracking. Regardless, now is the time to investigate these processes as well.
Bri Hatch is Chief Hacker at Onsight, Inc, and author of Hacking Linux Exposed and Building Linux VPNs. He's impressed with his recent experiment to see how many people are so bored at work that they're interested in watching a movie of the two year old daughter of a Linux Security author doing the Hula. Quite a turnout. Bri can be reached at bri@hackinglinuxexposed.com.
Copyright Bri Hatch, 2002.
This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772 on 21-May-2002.