Using TCP Flags to filter 3-Way Handshake using TCPDUMP / Wireshark
Matching TCP traffic with particular flag combinations can be a useful way of examining TCP conversations. Refer to the TCP State Machine to understand the context of these flags:
USING TCPDUMP TO FILTER ON FLAGS:
The flags are defined in the 14th byte of the TCP header.
+-+-+-+-+-+-+-+-+
|C|E|U|A|P|R|S|F|
|W|C|R|C|S|S|Y|I|
|R|E|G|K|H|T|N|N|
+-+-+-+-+-+-+-+-+
In the TCP 3-way handshakes, the exchange between hosts goes like this :
1. Source sends SYN
2. Destination answers with SYN, ACK
3. Source sends ACK
– If we want to match packets with only the SYN flag set, the 14th byte would have a binary
value of 00000010 which equals 2 in decimal.
# tcpdump -i eth1 ‘tcp[13] = 2’
– Matching SYN, ACK (00010010 or 18 in decimal)
# tcpdump -i eth1 ‘tcp[13] = 18’
– Matching either SYN only or SYN-ACK datagrams
# tcpdump -i eth1 ‘tcp[13] & 2 = 2’
We used a mask here. It will returns anything with the ACK bit set (thus the SYN-ACK combination as well)
Let’s assume the following examples (SYN-ACK)
00010010 : SYN-ACK packet
00000010 : mask (2 in decimal)
——–
00000010 : result (2 in decimal)
Every bits of the mask match !
– Matching PSH-ACK packets
# tcpdump -i eth1 ‘tcp[13] = 24’
– Matching any combination containing FIN (FIN usually always comes with an ACK so we either
need to use a mask or match the combination ACK-FIN)
# tcpdump -i eth1 ‘tcp[13] & 1 = 1’
– Matching RST flag
# tcpdump -i eth1 ‘tcp[13] & 4 = 4’
USING WIRESHARK:
By simply using “tcp.flags & [number]” (without the quotes), you can easily filter interesting parts of the TCP conversation (such as SYN, SYN/ACK, FIN and FIN/ACK.