dropwatch - discover where network packets are droppedSat 30 May 2020 by admin
Let's imagine situation where you experience network problem with dropping packets and you've no idea where the problem is located. So first of all prepare environment:
# iptables -A OUTPUT -p icmp -j DROP # ping -c 3 -W 1 22.214.171.124 PING 126.96.36.199 (188.8.131.52) 56(84) bytes of data. ping: sendmsg: Operation not permitted ping: sendmsg: Operation not permitted ping: sendmsg: Operation not permitted --- 184.108.40.206 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2012ms
forget what you've done and start digging with new approach call dropwatch. Dropwatch is a tool to monitor where in linux kernel packets are dropped. It's using kernel symbols to decode memory addresses into function name, start it using kernel symbols:
# dropwatch -l kas Initalizing kallsyms db dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 1 drops at nf_hook_slow+b0 (0xffffffff8178d6c0) 1 drops at nf_hook_slow+b0 (0xffffffff8178d6c0)
and here we have something related to netfilter. Function nf_hook_slow return
-EPERM when somethings is dropped in netfilter.
EPERM is translated into
1 value according to errno.h. Let's figure out how to get return value from this function. To do this we can use bcc tools, especially one of them called
# /usr/share/bcc/tools/trace -p $(pgrep ping) 'r::nf_hook_slow "%d", retval' PID TID COMM FUNC - 1890 1890 ping nf_hook_slow -1
got its return value
-1 is related to netfliter drops.