In my previous post about the IP Provider I got the following comment: “There is nothing unpleasant about the wonderfulness that is tcpdump! You’ll need to put a lot of work in to match tcpdump’s usefulness with Dtrace…”

That just sounds like a challenge. Bring it on! Can snoop or tcpdump do this?

root@ultra ~$ ./ip_whosent.d 
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 71 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 72.14.207.99: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 52 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 54 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 87 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 58 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 64 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 65 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 208.67.219.230: 644 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 637 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 72.14.207.99: 660 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 664 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 48 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 72.14.207.99: 40 byte packet on behalf of firefox-bin (PID: 1944)
^C

Here is the script:

#!/usr/sbin/dtrace -qs 



ip:ip:*:send
/execname != "sched"/
{ 
        printf("Packet sent to %s: %d byte packet on behalf of %s (PID: %d)n", 
                        args[2]->ip_daddr, args[4]->ipv4_length, execname, pid ); 
}

Oh but wait……. how about a full call stack on each sent packet? Just add a new line to the above script: stack();

root@ultra ~$ ./ip_sentstack.d 
Packet sent to 72.14.207.99: 84 byte packet on behalf of ping (PID: 2020)

              ip`ip_wput_ire+0x21f5
              ip`ire_send+0x1c9
              ip`ire_add_then_send+0x2b9
              ip`ip_newroute+0xa0a
              ip`ip_output_options+0x18c7
              ip`icmp_wput+0x44a
              unix`putnext+0x22b
              genunix`strput+0x1ad
              genunix`kstrputmsg+0x261
              sockfs`sosend_dgram+0x26e
              sockfs`sotpi_sendmsg+0x4a8
              sockfs`sendit+0x160
              sockfs`sendto+0x8e
              sockfs`sendto32+0x2d
              unix`sys_syscall32+0x101

Or check out one of the examples on the IP Provider wiki page (this is almost certainly by Brendan Gregg):

# ./ipio.d
 CPU  DELTA(us)          SOURCE               DEST      INT  BYTES
   1     598913    10.1.100.123 ->   192.168.10.75  ip.tun0     68
   1         73   192.168.1.108 ->     192.168.5.1     nge0    140
   1      18325   192.168.1.108    192.168.10.75  ip.tun0     20
   0         79   192.168.1.108 ->     192.168.5.1     nge0     92

Here is the script:

#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %3s %10s %15s    %15s %8s %6sn", "CPU", "DELTA(us)",
            "SOURCE", "DEST", "INT", "BYTES");
        last = timestamp;
}

ip:::send
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s -> %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_saddr, args[2]->ip_daddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

ip:::receive
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s elapsed,
            args[2]->ip_daddr, args[2]->ip_saddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

Can DTrace decrypt IPsec ESP payloads? No. Ok, so tcpdump isn’t dead yet, but the capabilities offered by DTrace are far deeper. I’ve got a ton of ideas more that I could put here, but don’t have time atm. DTrace for the win!