Summary: How can a program be exploitable by an attacker, even after it's been deleted?

Yes, it's been a while, and what better way to get back into the swing of things than with another contest. What are the rules, you ask? See the end of the article.

This week, we'll take a look at a successful and somewhat puzzling machine compromise. The machine in question was a production machine that had been up and running for about a year - one of those machines that had so much on it that you're afraid to ever reboot it, lest something not come back up.

The administrators did a great job of keeping the machine up to date. All vendor and software alerts went to their pagers, and they were amazingly fast at upgrading when any vulnerabilities in the installed software were found. The system had several hundred users, with fairly restrictive permissions on the system itself to prevent tampering, but they didn't watch user's actions much if at all.

Well, it happened that one user would connect to his shell account at his ISP now and then, and that shell server was compromised. Unfortunately, the user had the same password on both systems, and the cracker was able to log onto this extremely important machine because of it. The user never noticed that his account was being used from strange locations.

Now the good news is that, since the machine was kept very up to date, the cracker wasn't able to instantly leverage his shell access to get root access. He twiddled around all over the place trying to find something that could be attacked, and didn't find a thing. He too was likely monitoring vulnerability mailing lists and IRC channels, looking for anything that could be used, but nothing came of it.

A few months later, a bug was discovered in software that included a program named /usr/sbin/buggy[1]. This program was suid (set-user-id) root which means that when it is run it has an effective user id of root, regardless what user actually launched it. Such programs need to be very secure due to the elevated privileges.[2]

Getting back to /usr/sbin/buggy, this program turned out to have a vulnerability. When it was invoked with a carefully crafted environment, a buffer could be overflowed, and an attacker could run commands as root. This, obviously, is bad.

Naturally, the machine's administrators upgraded the package as soon as possible, and went about their business.

A few days later, they saw some interesting messages in their syslogs:

Dec  6 00:00 hst buggy[1930]: unable to parse ;g+gsz_zi.i|vDj2^qvCsOTD_b|SXD=DrmR^L+q
Dec  6 00:01 hst buggy[1957]: unable to parse ^F9LkKqg0NhYe[w|]]f-U=9d*f[gN!g^

The administrators had log monitoring tools running on this host, and any anomalies were sent to them in email. The next day they investigated the mysterious messages, and found (through a Google search, no doubt) that they were the logs that would be created if the older version of buggy were being exploited.

They quickly checked to make sure that their version was up to date, and it was. They tried the exploits themselves to see if they could reproduce the log messages:

jdoe$ ./exploit_buggy
==> Command to execute: /usr/bin/id
==> Setting environment variable
==> Running buggy
buggy: attempt to exploit vulnerability in version x.y.z detected.
buggy: this attempt has been logged.

# tail -500 /var/log/messages | grep buggy
Dec  6 09:31 hst buggy[20518]: attempt to exploit env bug by user jdoe foiled
Dec  6 09:31 hst buggy[20518]: time to get out a can of whoop-a**

So, the software was clearly not vulnerable to the exploit, yet the earlier logs seemed to indicate that it was. They left things alone for the night, and decided to check on it in the morning.

Sure enough, the next morning there were more similar logs

Dec  6 23:40 hst buggy[12113]: unable to parse j_D^}qu^PLcnKvqjpH`x_
Dec  6 23:49 hst buggy[12189]: unable to parse xpEhh|=wnQ~tL|8e^k}

They thought that, perhaps, there was a new bug in the software that wasn't publicly known yet. So, they decided to change the permissions on buggy:

# ls -la /usr/sbin/buggy
-rwsr-xr-x    1 root     root        29128 Nov 29 12:13 /usr/sbin/buggy
# chmod go-rx /usr/sbin/buggy
# ls -la /usr/sbin/buggy
-rws------    1 root     root        29128 Nov 29 12:13 /usr/sbin/buggy

Now the program could only be run as root, and should be safe from any users. The next morning, what should happen to be in their logs but yet more intrusions:

Dec  7 04:09 hst buggy[31203]: unable to parse bv,?cC!sFod`hys
Dec  7 04:54 hst buggy[32895]: unable to parse }hcj:j!Ul'kyaf$x}CI>|
Dec  7 04:54 hst buggy[32896]: unable to parse lv|3_t`avgfO3},y

What was going on? /usr/sbin/buggy could only be run by root. Presumably, there'd be no reason for the attacker to attempt an exploit if they were already running as root.

In a last ditch effort, they deleted the entire package.

# dpkg -P buggy
# ls -la /usr/sbin/buggy
ls: /usr/sbin/buggy: No such file or directory

There! Now that'll fix the problem, right?

Unsurprisingly[3], the next morning there were more logs, and lots of other changes that indicated someone had indeed been messing around the system as root, presumably by exploiting the buggy program to gain said root privileges.

So, the question is, of course, how did the user manage to exploit 'buggy', in spite of all the steps the administrators took?

• Upgrading to secure version
• Removing global execute permissions
• Removing the program itself

If you think you have an answer, write me an email (send to me, do not reply to the list software, please) with your thoughts. I'll send the closest, best, or most well worded answer a copy of Hacking Linux Exposed, 2nd Edition. I'll post the answer in December.[4]

NOTES:

[1] Real pathname removed to protect the innocent, and because I can't remember what it was any more

[2] Programs only need a suid or sgid bit when they need to have access to something not available to the invoking user. For example /bin/ping needs to have root access in order to send ICMP packets. You should never add a suid or sgid bit to a program unless you really really know the security ramifications of it.

[3] Otherwise it wouldn't be a very challenging contest, now would it?

[4] Yes, I'll have another post between now and then.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He's not keen on the idea of picking 4 potential baby names, knowing he'll only be using two. He's really hoping that tomorrow's ultrasound puts the baby gender issue to rest. If not, he'll be accepting baby name suggestions. Of course, how could a name make it on the short list without first determining if the domain is available... Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2003

This is the November 11, 2003 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.