Summary: The ability to use files seemingly deleted from the /proc directory helps savvy attackers avoid detection and remove incriminating files.

I was sitting on my laptop looking at a movie [1] of my daughter Reegen in her hula skirt on our trip to Hawaii this January. She and Allison, the daughter of one of our friends, were playing "Ring around the Rosey" and it was terribly cute.

Unfortunately, in a not-rare-enough multitasking mistake, I accidentally deleted the movie I was watching! I was in the /movies directory, and wanted to delete mvi_051*.avi and accidentally typed 'rm mvi_051 *.avi', wiping out all my movies! No! It's not fair! I hadn't made backups of the file yet and hadn't yet posted it to my Web site. A beautiful moment lost. What was I to do? [2]

No problem, lsof and /proc to the rescue. I had lsof (List Open Files, one of the must-haves in your UNIX administration and security toolboxes) show me the open files for the mplayer [3] process:

$ lsof -c mplayer | grep avi
mplayer 10153  bri   10r   REG 3,7 2545962  26708 /movies/hula.avi
(deleted)

This shows that mplayer (process 10153) has /movies/hula.avi opened on file descriptor 10. (The rest of the data describe the device it's on, size of file, etc....) The '(deleted)' at the end signifies that the file has been deleted from disk. However, Linux files aren't actually removed from the disk until all open file descriptors are closed and all hard links to the data are removed. Thus, the file was still there; I just couldn't get it by looking in the /movies directory because the hula.avi name had been removed.

However it's still possible to get at the file. All I needed to do was:

$ cp /proc/10153/fd/10 /tmp/hula_recoved.avi

The /proc filesystem is not an actual directory on disk like /usr or /home. Instead, /proc is a directory-based view of information the kernel makes available to you. The programs ps or top, for example, look in this directory for process ids and program names, and then presents them in a pretty form. The files and directories in the /proc/10153 directory refer to the process 10153 (mplayer). A quick list shows us:

$ ls -F /proc/10153/fd
0@  1@  10@  12@  2@  3@  4@  5@  6@  7@  8@  9@

$ ls -l /proc/10153/fd/10
lr-x------ 1 bri  hle     Apr 30 10:39 10 -> /tmp/reegen_hula.avi
(deleted)

The fd (file descriptor) directory has maps to the files open by the program. So you can see that there are many file descriptors open (0, 1, 2... 10, and 12), /proc tries to show information in the most useful UNIX-like way. Although it looks like /proc/10153/fd/10 is a symlink to the file '/tmp/reegen_hula.avi (deleted)', when you try to copy this file, it will give you the actual bits still on disk because the file hasn't been permanently removed from the hard drive.

So, what does this have to do with security?

One common trick malicious hackers use is to open a file and immediately delete it, such that the file is not visible on the machine to tools like find/locate/etc. This also means that, if the machine is rebooted, then the file disappears as well. Until the program stops, the file is still completely usable to itself. It may be a temporary storage space for lists of machines to compromise, copies of newly downloaded attack scripts, or captured passwords to be sent back to the attacker. Using deleted files is an easy method to avoid detection from most administrators, and automatically removes any incriminating files in the event an admin figure's something is amiss and kills the process or reboots the machine.

Next week, I'll show you a few other related /proc and lsof tidbits that can be useful at preventing people from using this trick against you. But for now, remember that if you delete something but still have it open, you have an alternative to misery. In the event that the deleted item is a cute movie of your daughter, you'll understand the need.

NOTES

[1]

[2] If this drive were an ext filesystem, I may have been able to recover all the files using e2undel () or other similar tools. Unfortunately I have most of my partitions formatted with ReiserFS. Fortunately, the rest of the movies were already backed up.

[3] Mplayer, a great Linux movie player, at

Bri Hatch is Chief Hacker at Onsight, Inc, and author of Hacking Linux Exposed and Building Linux VPNs. He really needs to sit down and post the last 6 months of pictures he's taken. At this rate, his daughter will be through college before her 2nd birthday pictures reach the Web. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002.

This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772  on 07-May-2002.