dropwatch - discover where network packets are dropped

Let’s imagine situation where you experience network problem with dropping packets and you’ve no idea where the problem is located. So first of all prepare environment:

# iptables -A OUTPUT -p icmp -j DROP
# ping -c 3 -W 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2012ms

forget what you’ve done and start digging with new approach call dropwatch. Dropwatch is a tool to monitor where in linux kernel packets are dropped. It’s using kernel symbols to decode memory addresses into function name, start it using kernel symbols:

# dropwatch -l kas
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at nf_hook_slow+b0 (0xffffffff8178d6c0)
1 drops at nf_hook_slow+b0 (0xffffffff8178d6c0)

and here we have something related to netfilter. Function nf_hook_slow return -EPERM when somethings is dropped in netfilter. EPERM is translated into 1 value according to errno.h. Let’s figure out how to get return value from this function. To do this we can use bcc tools, especially one of them called trace:

# /usr/share/bcc/tools/trace -p $(pgrep ping) 'r::nf_hook_slow "%d", retval'
PID     TID     COMM            FUNC             -
1890    1890    ping            nf_hook_slow     -1

got its return value -1 is related to netfliter drops.