Summary: A motto appropriate for both boy scouts and security admins: Always be prepared. When a vulnerability surfaces, it could mean the difference between remotely patching a hole and driving across town to your server room.

About twenty minutes after the OpenSSH vulnerability[1] was discovered, I started getting a number of messages from readers asking how to upgrade OpenSSH (http://www.openssh.com) safely. Luckily, I already had written an article on this very topic, assuming it'd come in useful at some point. So here goes!

Many Linux distros are starting to ship with the OpenSSH daemon. OpenSSH is a replacement for the insecure rlogin, rsh, rcp, ftp, x11, and telnet protocols that offer no encryption and are vulnerable to numerous insertion, hijacking, ip spoofing, and other attacks. OpenSSH also has encrypted port-forwarding features that make it indispensable at times.

Anyone with security in mind should have long since turned off telnet and friends, and should rely on ssh for logins only. The problem occurs when it's time to upgrade the ssh daemon. If your machines are in a server room miles away with only SSH access, the last thing you want to do is either break sshd during the upgrade or get in your car and go back to the office.

Most folks don't realize that you can kill off the sshd server process (the one that forks off copies to handle incoming connections) without killing off any existing connections. That means you can log in, kill off the server process, and still work on the system until you log out. Instead, most folks think you need to upgrade and reboot so the changes take effect. This is not the case. However, if you don't take a few precautions, then you could find yourself in the unfortunate situation where you've killed the old process and accidentally logged out (or, more likely, kill all sshd processes) before starting the new daemon.

The following is my extremely paranoid procedure to ensure that I can remain sitting in my comfy chair, legs propped on the desk, and a bottle of caffinated beverage within easy reach. Driving to the server room is for Windows administrators.

The following assumes your ssh etc files are in /etc/ssh, and that your sshd binary is in /usr/sbin. Salt to taste.

1. Establish a copy of your working sshd daemon on port 9999:

  server# mkdir /root/ssh-whoops
  server# cd /root/ssh-whoops
  server# cp -p /etc/ssh/* /usr/sbin/sshd  .
  server# /root/ssh-whoops/sshd -p 9999 -f /root/ssh-whoops/sshd_config

2. Connect to your machine on this port a few times to make sure it works, and give you a few login sessions in case of bad things later.

  home$ ssh MACHINE -p 9999
  home$ ssh MACHINE -p 9999
  home$ ssh MACHINE -p 9999
  home$ (repeat until paranoia satisfied)

3. Upgrade your OpenSSH software in whatever way you use.

  server#  rpm -F ...

  server#  apt-get update && apt-get upgrade

  server#  ./configure && make && make install

4. Find the normal sshd process id:

  server#  ps -ef | grep sshd
  root   10283 12684  0 Feb05 ?  00:00:05 /usr/sbin/sshd
  root   12684     1  0 Dec03 ?  00:01:21 /usr/sbin/sshd
  root   15446     1  0  4:15 ?  00:00:00 /usr/sbin/sshd -p 9999 -f ...
  root   15846 15446  0  4:16 ?  00:00:00 /usr/sbin/sshd -p 9999 -f ...
  root   15850 15446  0  4:16 ?  00:00:00 /usr/sbin/sshd -p 9999 -f ...
  root   16748 12684  0 Feb25 ?  00:00:00 /usr/sbin/sshd

The process you're looking for will be the daemon (i.e., the PPID is 1) that is *not* running with the '-p 9999' arguments. In the above listing, that'd be process #12684.

5. Kill the sshd server process:

  server# kill 12684

6. Wipe the sweat from your brow. It'll be ok.

7. Start the new ssh daemon (assuming you installed it in /usr/sbin where the original was).

  server# /usr/sbin/sshd

8. Log in from home and make sure everything's ok.

  home$ ssh -v MACHINE

9. Make sure that you see the version number you expect in the debugging output. For example, if you'd just upgraded to 3.1p1, you'd want to see a line like the following:

Remote protocol version 1.99, remote software version OpenSSH_3.1p1

10. If things aren't ok, then you still have a few login sessions open to use to fix the problems (perhaps the semantics of sshd_config has changed between versions) and also have the old version of the daemon available on port 9999, which you could connect to like you did in step 1 above, even if you accidentally log out of all your sessions. Fix the problems and go back to step 6. 11. When all problems are fixed, kill off the temporary sshd listening on port 9999 and nuke the old files:

  server# kill 15446
  server# rm -r /root/ssh-whoops

Using this method, you've got enough preventative measures that you should have no worries if the new sshd doesn't work right away. No need for a reboot and, more importantly, no need to leave your chair.

NOTES

[1] Affects OpenSSH 2.0-3.0.2. Suggest you upgrade to 3.1 pronto. OpenSSH 1.x versions are not vulnerable. SSH.com's software is also not vulnerable to this bug -- there's a first time for everything.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He thinks that anyone still using telnet as anything but a networking debugging tool should be banished. And even then, they should be using netcat for goodness sake. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002.

This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772  on 19-Mar-2002.