Comments on: How to use tcpdump on very busy hosts https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/ Fri, 10 Mar 2017 15:18:41 +0000 hourly 1 https://wordpress.org/?v=6.5.2 By: Roy https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-10967846 Fri, 10 Mar 2017 15:18:41 +0000 https://www.percona.com/blog/?p=6092#comment-10967846 Also in this same blog > https://www.percona.com/blog/2015/04/10/measuring-impact-tcpdump-busy-hosts/

]]>
By: Roy https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-10967845 Fri, 10 Mar 2017 15:17:42 +0000 https://www.percona.com/blog/?p=6092#comment-10967845 I really suggest to store the raw log (I can intercept easily 200Mbit/s with very low cpu usage) and later process it.

]]>
By: Erik Brakkee https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-10379125 Wed, 14 Jan 2015 08:17:34 +0000 https://www.percona.com/blog/?p=6092#comment-10379125 I have been using the performance optimization with the tcp flags filtering in tcpdump for some time. Yesterday however, I found out that I could not capture any packets anymore on a number of hosts (not all). All of these hosts are running Centos 6 (various versions). I could not figure out what the problem was so had to remove the TCP flags again.

]]>
By: Pascal Schmiel https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-886916 Thu, 09 Feb 2012 10:43:22 +0000 https://www.percona.com/blog/?p=6092#comment-886916 You can improve the rate even further if you first capture the traffic to a file using tcpdump’s “-w” option and parse it later with “tcpdump -r”.

Using /usr/sbin/tcpdump -i eth1 -c 100000 -s0 -x -nn -q -tttt ‘port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2’ | /usr/bin/mk-query-digest –type=tcpdump :

100000 packets captured
195794 packets received by filter
95664 packets dropped by kernel

And now using /usr/sbin/tcpdump -i eth1 -c 100000 -s0 -w /tmp/snoopfile ‘port 3306 and tcp[1] & 7 == 2 and tcp[3] & 7 == 2’; /usr/sbin/tcpdump -r /tmp/snoopfile -s0 -x -nn -q -tttt | /usr/bin/mk-query-digest –type=tcpdump :

100000 packets captured
100286 packets received by filter
220 packets dropped by kernel

]]>
By: Baron Schwartz https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805697 Tue, 26 Apr 2011 13:28:16 +0000 https://www.percona.com/blog/?p=6092#comment-805697 I just used this trick, worked like a charm.

207815 packets captured
214152 packets received by filter
6324 packets dropped by kernel

]]>
By: Anthony DeRobertis https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805102 Tue, 19 Apr 2011 16:13:39 +0000 https://www.percona.com/blog/?p=6092#comment-805102 Another approach, if you have the hardware: enable port mirroring on your switch, and capture the traffic on an otherwise idle host with enough disk bandwidth to do so.

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805009 Mon, 18 Apr 2011 22:01:43 +0000 https://www.percona.com/blog/?p=6092#comment-805009 Devananda .. right I missed that.

]]>
By: Devananda van der Veen https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805008 Mon, 18 Apr 2011 21:58:39 +0000 https://www.percona.com/blog/?p=6092#comment-805008 erkules, “dst port 3306” won’t work because you must also capture the reply from mysqld, where the source port is 3306 and the dest port is unknown (matches & 7 == 2).

In some cases, I’m sure that host filtering would work fine, eg when you want to watch a specific host, but in some situations I’ve faced it does not work well. Honestly, I’m not sure why the packet drops were so high when filtering just a few hosts, perhaps the CPU overhead is higher when applying a host IP filter versus a port bitmask filter?

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805007 Mon, 18 Apr 2011 21:54:25 +0000 https://www.percona.com/blog/?p=6092#comment-805007 Ahh of course no effect is wrong. At least of the ports got to be 3306 (port=3306) the AND check for that dst/src-port is fulfilled anyway (3306) and the other one got to fullfill &7 == 2.

So one could use instead “dst port 3306” to match mysql server and do some portfiltering on tcp[0,1] I would prefer hosts.

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805006 Mon, 18 Apr 2011 21:46:49 +0000 https://www.percona.com/blog/?p=6092#comment-805006 On this example it has no effect, as port 3306 is more precise. tcp[0,1] sourceport and tcp[2,3] dstport (rfc 793).

]]>
By: Devananda van der Veen https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805005 Mon, 18 Apr 2011 21:42:05 +0000 https://www.percona.com/blog/?p=6092#comment-805005 tcp[x] indicates the x’th octet in the TCP header, starting from 0. The first two octets are the source port, second two octets are the destination port. So my understanding is that this expression says: the lower octet of each source and dest port, when AND’ed with the bitmask ‘111’ result in a value of ‘010’, or the number 2. Note that “3306 & 7 == 2” is a true statement.

See http://www.tcpdump.org/tcpdump_man.html for more in-depth info on TCP header structure.

]]>
By: Baron Schwartz https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-805003 Mon, 18 Apr 2011 21:04:49 +0000 https://www.percona.com/blog/?p=6092#comment-805003 I’m not very fresh on my TCP headers and I don’t remember what’s in various positions. What is the net effect of this expression?

tcp[1] & 7 == 2 and tcp[3] & 7 == 2

]]>
By: Devananda van der Veen https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806385 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806385

erkules, “dst port 3306″ won’t work because you must also capture the reply from mysqld, where the source port is 3306 and the dest port is unknown (matches & 7 == 2).

In some cases, I’m sure that host filtering would work fine, eg when you want to watch a specific host, but in some situations I’ve faced it does not work well. Honestly, I’m not sure why the packet drops were so high when filtering just a few hosts, perhaps the CPU overhead is higher when applying a host IP filter versus a port bitmask filter?

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806386 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806386

Ahh of course no effect is wrong. At least of the ports got to be 3306 (port=3306) the AND check for that dst/src-port is fulfilled anyway (3306) and the other one got to fullfill &7 == 2.

So one could use instead “dst port 3306″ to match mysql server and do some portfiltering on tcp[0,1] I would prefer hosts.

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806387 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806387

On this example it has no effect, as port 3306 is more precise. tcp[0,1] sourceport and tcp[2,3] dstport (rfc 793).

]]>
By: Devananda van der Veen https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806388 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806388

tcp[x] indicates the x’th octet in the TCP header, starting from 0. The first two octets are the source port, second two octets are the destination port. So my understanding is that this expression says: the lower octet of each source and dest port, when AND’ed with the bitmask ’111′ result in a value of ’010′, or the number 2. Note that “3306 & 7 == 2″ is a true statement.

See http://www.tcpdump.org/tcpdump_man.html for more in-depth info on TCP header structure.

]]>
By: Baron Schwartz https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806389 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806389

I’m not very fresh on my TCP headers and I don’t remember what’s in various positions. What is the net effect of this expression?

tcp[1] & 7 == 2 and tcp[3] & 7 == 2

]]>
By: erkules https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806384 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806384

Devananda .. right I missed that.

]]>
By: Anthony DeRobertis https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806377 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806377

Another approach, if you have the hardware: enable port mirroring on your switch, and capture the traffic on an otherwise idle host with enough disk bandwidth to do so.

]]>
By: Baron Schwartz https://www.percona.com/blog/how-to-use-tcpdump-on-very-busy-hosts/#comment-806336 Mon, 18 Apr 2011 07:00:01 +0000 https://www.percona.com/blog/?p=6092#comment-806336

I just used this trick, worked like a charm.

207815 packets captured
214152 packets received by filter
6324 packets dropped by kernel

]]>