Summary: This week, Bri presents our readers with a challenge to help solve the man-in-the-middle attack.

Recently, a machine on which I have an account had several accounts cracked. I hadn't actually logged into the machine in a few months, but one of the admins sent me an email asking me to help track down the problem.

When logging in, several users reported seeing themselves already logged in from strange locations or running funny processes. Most of these folks are generally security-conscious, use strong passwords, and don't fall for the standard social engineering tricks. Because the admin is good and paranoid, the machine is always kept up to date on patches and ssh is the only way to login, meaning the traffic is all encrypted. I suspected some vulnerability that wasn't yet public knowledge on the 'Net.

I went to log in and was greeted with this:

home$ ssh server
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA1 host key has just been changed.
The fingerprint for the RSA1 key sent by the remote host is
48:45:0b:e7:0b:bb:70:ac:8e:94:93:7f:33:44:10:c6.
Please contact your system administrator.
Add correct host key in /home/bri/.ssh/known_hosts to get rid of this message.
Offending key in /home/bri/.ssh/known_hosts:17
RSA1 host key has changed and you have requested strict checking.
home$

The ssh host key had changed. Any time you see this, you should suspect the worst. However, since we've had a few ssh vulnerabilities recently, I figured that the admin had accidentally wiped out the original ssh keys in /etc/ssh and needed to make new ones. Heck, I've done it myself.

So I commented out the old key (line 17 in my ~/.ssh/known_hosts file) and re-ssh'd in, accepting the new key.

Before I tracked down the intrusion, I figured that I'd see if I could get the original ssh host key from backups so I (and others) wouldn't get that annoying message each time we login. I went into /etc/ssh, where the keys are kept, and saw the following:

server$ ls -l /etc/ssh*
-rw-r--r--    1 root     root    1412 Jun 12 12:10 ssh_config
-rw-------    1 root     root     526 Jun 10 09:44 ssh_host_key
-rw-r--r--    1 root     root     330 Jun 10 09:44 ssh_host_key.pub
-rw-r--r--    1 root     root    1216 Jun 12 12:13 sshd_config

I was trying to see if I'd made a directory named /etc/ssh.bak or similar. (I tend to leave backups of /etc/ssh around out of laziness). I was not surprised to see no backup ssh directory, but I was surprised to see that the host key had not been changed recently -- Jun 10th was when the machine was installed in the first place. Checking 'ls -c' output (to see the inode change time) also showed the Jun 10th date.

I compared (visually) the public host key file on the server with line 17 of my ~/.ssh/known_hosts on my home machine; they were the same, but if they really were the same, I shouldn't have gotten that warning. I tried ssh'ing in again to see if I was going crazy, but no; in fact, the key being presented was different from server:/etc/ssh/ssh_host_key.

I stopped and restarted the sshd daemon in debug mode under ptrace and verified that it was reading the correct key files in /etc/ssh. Still the wrong key was being presented.

At this point, I was pretty convinced I'd solved both the ssh problem and the cause of the cracked accounts. To test my theory, I ran a copy of sshd as follows:

server$ sudo /usr/sbin/sshd -d -p 4321

This started an ssh server on port 4321. I uncommented line 17 of my ~/.ssh/known_hosts, putting the file back the way it had been for the last nine months. I then connected to the server from home as follows:

home$ ssh -p4321 server 'echo connected to `hostname`'
connected to server
home$

As you see, my ssh connection to port 4321 was established successfully and the lame 'echo' command ran. But there was no complaint about the host key being different. The key on line 17 of my ~/.ssh/known_hosts matched the key presented by the ssh server on port 4321. /usr/sbin/sshd was using the key from /etc/ssh as it should.

So, what was wrong? I'll give you the answer in two weeks. If you think you have it figured out, drop me a line. I'll send the first person who gets it right a postcard and immortalize them in the follow- up article. Extra points if you can describe how to trace down the culprit.

Bri Hatch is Chief Hacker at Onsight, Inc, and author of Hacking Linux Exposed and Building Linux VPNs. He's very sad that he's been taken off the Microsoft recruiting 'do not call' list, and wonders what he needs to do to get blacklisted again. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002.

This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772  on 16-Apr-2002.