Thursday, October 24, 2019

Using Wireshark on a remote host

In a large environment, troubleshooting problems with network packet traces usually means you're logged into a remote host running tcpdump.  Even after you develop some skill with pcap-filter syntax, wielding tcpdump is clunky and it usually looks like you're trying to view The Matrix encoded.

There are other console based tools like tshark, but few of them are as useful and as user-friendly as Wireshark which can render and parse network packets in an extremely readable and comprehensive fashion.

The problem is that Wireshark is a graphical interface.  Running it on a remote host means you'll have to install it and all supporting dependencies and libraries on the remote host and then ssh X tunneling it back to your desktop.  For many reasons, this may not work well.  Or, you may not even be able to install Wireshark on the remote host for any number of reasons.

One workaround used by a lot of people is to capture some network output with tcpdump writing to a file, then fetch that capture file to your desktop and open it up in Wireshark.  It's definitely handy that pcap is so portable that this is possible, but this method lacks the ability to watch network traffic in real-time.

So how can you achieve the holy grail and use Wireshark locally on your desktop to watch live traffic on a remote host?

Enter socat - Multipurpose relay.

The socat utility is a swiss army knife of basically all possible types if input/ouput.  One of its supported i/o types is named pipes.

In short, we can use socat as the middleman to read from a remote named pipe to a local named pipe. Then, we take advantage of Wireshark's ability to read right from a named pipe and read that local named file.

Here's the steps using the example username jsmith, example remote host name srv1, and example network interface name eth0.

On the remote host:

  1. Create a temp dir for your named pipe file.
    • sudo mkdir /tmp/fifo
    • sudo chown jsmith /tmp/fifo
    • sudo chmod 700 /tmp/fifo
  2. Create the named pipe
    • sudo mkfifo /tmp/fifo/pcappipe
  3. Kick off tcpdump, writing to that pipe.
    • sudo tcpdump -i eth0 -s 0 -U -w /tmp/fifo/pcappipe not port 22
Notice the temp dir permissions.  You need to be able to read the named pipe as the non-root user with which you're going to use to log in.

Also notice the pcap filter 'not port 22'.  You can alter this of course, but if you don't specifically exclude your ssh traffic, tcpdump is going to pick up all of the traffic from you being logged in as well as the part where we remotely read from the named pipe which takes place over ssh.

Next, on your local desktop, run socat like so:

socat -b 67108864 \
    EXEC:"stdbuf -i0 -o0 -e0 ssh -x -C -t srv1 cat /tmp/fifo/pcappipe",pty,raw \

This tells socat to ssh into the remote host and cat the named pipe (sending the data to STDOUT).  It reads from that and writes it to the named pipe file in your home directory.

The buffer tuning was important to making it as live as possible as well as more stable.  Plus, this can be somewhat of a brittle process and socat can end up crashing easily.  The buffer tuning helps make things much more stable and reliable.

Next, run wireshark, as root.

sudo wireshark -s 0 -k -i /home/jsmith/localpcappipe


Normal ssh rules apply.  So, if you can't ssh directly to your remote host, configure your .ssh/config file accordingly.

I need to tunnel through an intermediary jump host as well, so this is what I do in my .ssh/config file:

Host srv1
    ProxyCommand           ssh jumpsrv1 /usr/bin/nc %h 22
    User                   jsmith
    IdentityFile           ~/.ssh/id_rsa
    Compression            yes
    PubkeyAuthentication   yes
    Port                   22
    Protocol               2
    EscapeChar             none
    ServerAliveInterval    30

(I know I know, there's a new ProxyJump directive...  I don't change my .ssh/config that often.)