Summary: /bin/rm doesn't mean remove, it means unlink, and it has security repercussions.

In a previous article[1] I described a machine compromise that initially would seemed to have been impossible. A vulnerable suid root program, /usr/sbin/buggy, was upgraded to a non vulnerable version, and yet crackers still were exploiting it. In fact, even after the program was removed entirely, it was still being exploited.

So, how can it be that a program can be accessed even after it's been removed? The problem is that it wasn't removed.

When we access a file on disk, we do so by providing a filename for it. The filesystem uses filenames, which are simply entries in directories, to look up the file's inode. An inode is a uniq numeric identifier, essentially a file number, which identifies the file on that filesystem. You can see the inode with the -i option to ls:

$ ls -il /bin/cat
32689 -rwxr-xr-x   1 root   root  13912 Sep 17 06:54 /bin/cat

That first number in the second line is the inode number, here 32689. Note also the third field in the line, '1'. That's the link count, the number of directory references to the file. Now, compare /bin/cat to, say, /bin/gzip:

$ ls -il /bin/gzip
32755 -rwxr-xr-x   4 root   root  13912 Apr 19 04:43 /bin/gzip

Note that link count of 4 in this case. How can we find the other filenames that point to this same file? First, let's determine which filesystem /bin lives on, and then use find:

$ cd /bin

$ df .
Filesystem          1K-blocks     Used Available Use% Mounted on
/dev/hda5            10766920  8325216    1894740  82% /

$ find / -inum 32755 -xdev -ls 2>/dev/null 
 32755   56 -rwxr-xr-x   4 root   root  13912 Apr 19 04:43 /bin/gzip
 32755   56 -rwxr-xr-x   4 root   root  13912 Apr 19 04:43 /bin/gunzip
 32755   56 -rwxr-xr-x   4 root   root  13912 Apr 19 04:43 /bin/zcat
 32755   56 -rwxr-xr-x   4 root   root  13912 Apr 19 04:43 /bin/uncompress

Here we can see that the four files, /bin/gzip, /bin/gunzip, /bin/zcat, and /bin/uncompress are four different filenames, pointing to the same actual file on disk, inode number 32755.[2]

We call these hard links -- when a file on disk is available via separate filenames. Compare these to symlinks/softlinks, which are separate files that point to a filename, rather than an inode:

$ ls -il /bin/nc /bin/netcat
32502 -rwxr-xr-x   1 root   root  19352 Oct 13 15:20 /bin/nc
51222 lrwxrwxrwx   1 root   root      2 Oct 13 15:20 /bin/netcat -> nc

Now anyone can make a hard link or symlink, assuming they have write permission to a directory. While symlinks can cross filesystem boundaries (they point to file names, not inodes) a hard link can only be created on the filesystem where the target file resides.

Thus to create a hard link to /usr/sbin/buggy, the attacker needed to have write access to some directory that was on the same filesystem as /usr/sbin/buggy. Unfortunately, this machine was created with one big huge root (/) filesystem, which contained all the directories, including /tmp and /home.

So what happened? The attacker created a link to /usr/sbin/buggy long before anyone knew it had a vulnerability:

attacker$ cd $HOME
attacker$ ls -il /usr/sbin/buggy
18824 -rwsr-xr-x   1 root   root  19352 Feb  5 08:12 /usr/sbin/buggy

attacker$ ln /usr/sbin/buggy mybuggy.hardlink
attacker$ ls -il /usr/sbin/buggy mybuggy.hardlink
18824 -rwsr-xr-x   2 root   root  19352 Feb  5 08:12 /usr/sbin/buggy
18824 -rwsr-xr-x   2 root   root  19352 Feb  5 08:12 mybuggy.hardlink

Note how the link count has changed to 2 now -- this file on disk is now referenced by two filenames.

Later on, when the bug in /usr/sbin/buggy came to light, the administrators upgraded the package. The upgrade automatically deleted the old /usr/sbin/buggy and then copied the new one from the binary package into /usr/sbin. But when the upgrade deleted /usr/sbin/buggy, the kernel merely decremented the link count (from 2 to 1) of that inode, and didn't actually free up the disk space, because the file was still referenced by a different filename on the filesystem. The original program still existed on disk, with the original suid permissions.

attacker$ ls -il /usr/sbin/buggy mybuggy.hardlink
45110 -rwsr-xr-x   1 root   root  17889 Nov 29 12:13 /usr/sbin/buggy
18824 -rwsr-xr-x   1 root   root  19352 Feb  5 08:12 mybuggy.hardlink

That first file is the new, non-vulnerable /usr/sbin/buggy program. The second is the old version, which the attacker had linked, and is still hanging around, and which he used to gain root access periodically.[3]

So, how should the administrators have known that this link was still hanging around? There are several precautions that would have indicated or prevented the problematic situation:

• Many file integrity tools include link count as one of the attributes that they watch for. These would have noticed the link count go up by one, and they could have gone searching for the new link.

• If they ran suid program scanners, they would have picked up this 'new' suid program.

• If they'd put user-accessible directories on separate partitions, then the attacker would not have been able to create a hardlink in the first place. I prefer at a minimum to have /tmp, /var/tmp and /home on separate partitions for this reason. It's always best to have separate partitions for system use vs user use.

Several readers had other ideas of how this could have been caused. I'll cover some of them in following articles, as they are all interesting.

Congrats go to Jim Richardson of Seattle, WA for solving the contest, thus winning a signed copy of Hacking Linux Exposed Second Edition.

NOTES:

[1] See http://www.hackinglinuxexposed.com/articles/20031111.html.

[2] -xdev was used to make sure we didn't look for files with inode number 32755 on other filesystems, such as /home for example. An inode is uniq on a filesystem, not across separate filesystems.

[3] Yeah, it would have made much more sense for the attacker to put other backdoors into the system. I'm just concentrating on this one hole because anything else would be extraneous.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He was pleasantly surprised that the contest winner was a fellow Seattlite. Besides, then he was able to save money on postage. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2003

This is the December 14, 2003 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.