Summary: Tracking down your network daemons is extremely easy if you use the right tools.

Last week a reader asked the following question:

"I'm having trouble tracking down a process that's running on my machine. When I run netstat -a, I see lots of things that should be there, but also the following:

  $ netstat -a
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address    Foreign Address   State
  tcp        0      0 localhost:imaps  *:*               LISTEN
  tcp        0      0 *:smtp           *:*               LISTEN
  ...
  tcp        0      0 *:8577           *:*               LISTEN
  ...

I can't figure out what that is on port 8577. Any ideas?"

My first reaction is almost universally to connect and see if it says anything useful. So I had him pull out netcat[1] to check the header:

  $ nc localhost 8577
  RFB 003.003

The server process responded with the RFB line. That's what you get when you connect to a VNC server, just like you'd expect SSH-1.99-OpenSSH.... for an SSH server that supports v1 and v2. Most network daemons will say something when you connect. You can usually match the output (aka the banner) to a protocol or process.

However there are two other ways that are even more direct, that can tie the network port to the actual process that is listening. For example on the machine running the VNC server on port 8577, there was no process called vncserver running on the machine.

So, we turn to two tools: netstat, and our old friend lsof.

Netstat has a -p option that will show you the name and pid (process id) of the local process that is associated with a connection. If you're running as root, you can see the processes for all users. If you're a normal user, you can only see the processes that are running as you.

So, re-running the netstat above with -p would have yielded

  # netstat -ap
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address    Foreign Address   State
  tcp        0      0 localhost:imaps  *:*               LISTEN  277/stunnel
  tcp        0      0 *:smtp           *:*               LISTEN  394/master
  ...
  tcp        0      0 *:8577           *:*               LISTEN  57283/bash
  ...

  # ps -fc 57283
  UID    PID  PPID  CLS PRI STIME TTY  STAT  TIME CMD
  doug 57283     1    -  29 10:30   ?  S     0:00 /home/doug/bin/bash

So here we can see that /home/doug/bin/bash (a copy of vncserver under a different name to avoid detection) is the process listening on port 8577, and it can be killed and doug scolded as appropriate.

Alternatively, you can use lsof to do the same thing

  # lsof -i tcp:8577
  COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
  ssh     57283 doug    3u  IPv4   31740       TCP *8577 (LISTEN)

If you're interested in what the process is doing, you can use lsof -p to see it's open files, strace -p to watch it's system calls, or ltrace -p to watch library calls. If you're comfortable with full blown debugging, gdb can attach to it and give you all the control you could want.[2]

NOTES:

[1] You could use telnet too, but why?

[2] All of these actions require that you're root unless you own the process.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He likes to keep the number of open ports on his machines to a minimum. All his machines really need is XOR-Telnet for ultimate session security. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2003

This is the May 15, 2003 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.