Summary: Locking out old and suspect users is a good thing, but it's imperative that you keep a log of locked users to be able to investigate anything they've left behind.

Warning: Anyone who hasn't heard, the latest BIND bug has come to light. This time it affects the 4.x and 8.x streams. ISC has not released patches to the public yet, in spite of the fact that they have to their preferred (paying) users. You have three options: wait around until they make good on their promise to support the public, upgrade to BIND 9.x, or better yet switch to something else. Now's an excellent time to switch to DJBDNS. See http://www.hackinglinuxexposed.com/articles/20020716.html for a multi-part article to show you how you can transition with no downtime whatsoever.

Ok, now onto this week's column.

Last week I extolled the virtues of keeping an audit trail. This actually came up because a reader was confused about some processes that were running under a no-longer existing userid:

  $ ps -ef |grep john
  1091     23699 20893  0 Oct 3 ?        00:05:24 /tmp/.j/john n88o9
  jdoe      8430 23702  0 10:07 pts/7    00:00:00 grep john

The first entry is a john-the-ripper[1] process running under userid 1091. The second is simply the grep command run by jdoe. Checking /etc/passwd to see who user 1091 reveals nothing, unfortunately:

  $ grep 1091 /etc/passwd
  $

At some point the user had been deleted[2] from the system, but his processes remained. Not good. Unfortunately, they had no records of who originally used this account. There are many ways that a user can be deleted, be it manual changes or using the deluser program. Deluser removes entries from /etc/passwd and /etc/shadow entirely, rather than just locking the account by giving it a locked password.[3] It takes default settings from /etc/deluser.conf, which can be overridden by command-line arguments. The settings allow you to decide how you want files owned by the user to be handled.

In general, it's a good idea to remove the files, because you never know what they may have hidden around. You do need to be careful, of course. Say a user was maintaining part of your website, you wouldn't want to delete those files blindly. Thus many administrators will not use the delete files options for deluser and will delete them manually, to make sure important files remain.

So, how can you determine who used to be userid 1091? If you don't delete any files when you lock accounts, you can check /home itself:

  # find /home -maxdepth 1 -nouser -ls
  drwxr-xr-x   22 3775     users        4096 Oct 16 20:41 chris
  drwx--x--x   42 1120     users        4096 Nov 12 10:39 maddie
  drwxr-xr-x    3 1843     users        4096 Jul 18  2001 reegen
  drwxr-xr-x    3 1354     users        4096 Jul 19  2001 ryan  

Here you can infer the original username since it usually matches /home/directoryname. Unfortunately, the account-locking procedure on this machine is to nuke the home dir, so no dice here.

But, how about non-home directory files, for example mail spools which are named after their owner?

  # find /var/mail -nouser -ls
  -rwxr-xr-x    1 3775     users       83712 Jan 23  5:20 chris

Nope, still no userid 1091. Looks like the other files were deleted too. An exhaustive search of the filesystem only turned up the password cracking files in /tmp/.j, and none of those indicated who actually owned them.

If we knew the username, we could search the machine for tarballs named username.tar.gz, username.tgz, username.tar.bz2, and so on, in hopes that the original files owned by the locked user are tarred up somewhere. The name of the file should indicate the username, but the data contained within it should also do the trick. Even if it didn't have the username in the filename itself, we can do a file listing of the tarball:

  $ tar tzvf unknown_user.tar.gz
  -rw------- bob/1009        175 2001-12-26 23:34:54 home/tom/.bash_history
  -rw------- bob/1009        668 2001-09-12 19:06:27 home/tom/.viminfo
  -rw------- bob/1009        614 2001-09-12  3:21:16 home/tom/.bashrc
  ...

and here we'd see that 'bob' was the original owner. Unfortunately, no luck finding an archive on the machine. They did not use --backup in general when running deluser. (This has been corrected.)

So, the filesystem itself led nowhere. However we still had other places we could look, such as the system logs. We searched for the userid 1091 in all the logs, hoping to correlate the userid with a username. Perhaps a retrieval of email via POP, a login with SSH, etc. Unfortunately, their log rotation scripts only kept the last three weeks of logs, which meant older logs were lost forever.

On a whim, I thought about checking for the userid in the email spools of local users. Since the system uses Postfix, and it logs the userid (but not username) of each mail sent from it, there was a chance the user had sent email to other folks on this system that was still stored.

  # cd /var/mail
  # grep "from userid 1091" *
  john:Received: by localhost (Postfix, from userid 1091)
  sandy:Received: by localhost (Postfix, from userid 1091)
  sandy:Received: by localhost (Postfix, from userid 1091)
  ...

Huzzah! It seemed that this user had, at some point, sent emails to other folks on this machine. We merely pulled up the email headers that originated from userid 1091 and checked to see who they were from:

  ...
  Received: by localhost (Postfix, from userid 1091)
        id C823155F13; Tue, 18 Sep 2001 16:59:03 -0700 (PDT)
  ...
  From: mystery_user

The "From:" lines all matched, and the recipients indicated that they did believe that these emails were legitimate because they were part of continuing conversations with the locked individual.

So, we were eventually able to determine with some certainty who the locked individual was, but it was painful and time consuming.

I have a very simple trick I employ to be sure I can always match up a userid with an account, even if the account is locked. Rather than deleting the /etc/passwd entry, you instead prefix it with a minus sign, ala

  $ grep '^-' /etc/passwd
  -frank:x:3005:2000:Frank En Stein:/home/frank:/bin/poponly
  -joel:x:3007:2000:Joel Scambray:/home/joel:/bin/bash
  -amar:x:3012:2000:Amar B:/home/amar:/bin/bash

None of these users actually exist - the minus sign means 'remove this user/userid from the list' by abusing a bit of NIS trickery:

  $ finger frank
  finger: frank: no such user

Yet you can easily see who the user was, their home directory, and shell. Should you need to re-enable the user later, you have their preferred shell and everything at your fingertips.

Unfortunately, this method is not compatible with deluser, so you need to either forgo deluser and perform file/home directory deletion yourself, or you need to re-add the line back in when you're done. However it provides a simple trail that will let you know who used to be on the system. You can even include extra data in the GECOS field if you wish:

  $ grep '^-' /etc/passwd
  -bob:x:4204:200:Bob, LOCKED 2002/12/31, left company:/home/bob:/dev/null

Unfortunately the question remained - how did the user keep his access to the system? Something for next week.

NOTES:

[1] http://www.openwall.com/john/

[2] This machine is a public shell server, and has many userids greater than 1091. Users are deleted every so often for breaking the AUP, as this one must have. You might argue that a cracker broke root and started using 1091 because it doesn't exist, but that proved not to be the case as we learned later.

[3] This is a good thing. There are many ways a user can maintain access to their account even without knowing their password.

[4] or username.tar.gz, username.tar.bz2, etc, depending on your system.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He offers Linux accounts to anyone he trusts. The average number of accounts on his machines is somewhere between 2 and 3. Counting his own, and root. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002

This is the November 13, 2002 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.