Using pcap filters with pktcap-uw on ESXi

A feature that was introduced in vSphere 7 is the ability to use pcap (tcpdump) filters with pktcap-uw. One of the use cases where this can be useful is filtering for TCP control bits.

Sometimes when you don’t have access to the guest operating system of a VM you still want to gain insight in what network traffic is passed through its network interface. You could already use the pktcap-uw tool to do this, but the filtering options were rather limited.

Having the ability to use pcap filters you can create a capture filter that only monitors the TCP control bits (SYN, FIN, RST). This will dramatically reduce the number of captured packets, with the added benefit that you can leave the capture running for much longer and not having it fill up your disk in no-time. To further explain what to such a filter does, consider this TCP diagram.

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |        Destination Port       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Sequence Number                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Acknowledgement Number                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Offset |  Reserved |U|A|P|R|S|F|             Window            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |         Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Options                      |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              Data                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

To filter for just the control bits, you need to select the 13th byte and use an AND-operation (&) to mask the relevant bits (R|S|F). When used in a command this would look something like this:

pktcap-uw –switchport 12345678 –rcf “tcp[13] & 7 != 0”

To execute a capture in the background, you can use the ampersand character & at the end of a line. When you use pktcap-uw over a SSH connection you probably want to combine this with ‘nohup’ to prevent the process from being terminated when your SSH session disconnects. This results in the following command:

nohup pktcap-uw –switchport 12345678 –rcf “tcp[13] & 7 != 0” -o /vmfs/volumes/datastore1/mycapture.pcap &

To stop the capture you can use the instructions as listed in the VMware KB article 2051814.

kill $(lsof |grep pktcap-uw |awk ‘{print $1}’| sort -u)