Every time a new application integrates static DTrace probes there is much rejoicing… and I’m glad for that, but I’m afraid a lot of people think that DTrace isn’t useful in userland without probes being baked into the code. This couldn’t be further from the truth, in fact, putting static probes in your code is, 90% of the time, just a convenience.

If you don’t believe me, try this script on any application you like:

#!/usr/sbin/dtrace -s

#pragma D option flowindent

pid$target:a.out::entry
{
}
pid$target:a.out::return
{
}

Make that executable and run it like this:

root@ultra tmp$ ./pidflow.d -p `pgrep e16`
dtrace: script './pidflow.d' matched 3040 probes
CPU FUNCTION                                 
  1  -> EDebug                                
  1  <- EDebug                                
  1  -> TimersRun                             
  1    -> EDebug                              
  1    <- EDebug                              
  1    -> MenusTimeout                        
  1      -> ecore_list_goto_first             
  1      <- ecore_list_goto_first             
  1      -> _ecore_list_goto_first            
  1      <- _ecore_list_goto_first            
  1      -> ecore_list_next                   
  1      <- ecore_list_next                   
  1      -> _ecore_list_next   
...

In the above example your seeing functions within the Enlightenment DR16 window manager (best WM ever!!!!). There are no probes within E, I compiled this build months ago, there is no trickery involved. You could use this script on Apache2, MySQL, Oracle, you name it.

But how does this work? How does DTrace know what the internal functions are? Symbols my friend. If you’ve ever used nm you’ll know that symbols (of which many are functions) are visible, and DTrace uses these as probe functions. Each function has an entry probe and return probe, plus a variety of offset probes although typically entry/return is all you care about.

The example above uses the PID provider (in the form “pid123”, or in our case “pid$target” where $target is the PID fed in via the command line -p argument), and the “a.out” module. The module name “a.out” is actually a synonym for the binary name, so I could have used “e16” instead of “a.out”, but I use the latter to be generic. If you we didn’t specify the module name it would trace not only the e16 calls but also those of its shared libraries! This is handy if you want to watch, in the case of a window manger like Enlightenment, calls go in and out of E, X, or shared libraries. The probefunc isn’t specified here, so we see them all. When the braces (action) are empty DTrace simply traces on probefunc name, and using the flowindent option DTrace formats the output with indents to indicate entry and return. (Prop’s to Derek Crudgington for turning me into a flowindent fanatic.) Super handy!!!

But wait, there’s more! DTrace can look inside arguments passed in to these functions! For simple arguments like a char pointer you can examine it using a copyin operation like “copyinstr(arg1)”, assuming that arg1 (second argument to the function) is a char array. Things only get tricky when you examine structures, because DTrace doesn’t know what the structure looks like, this is where “Translators” come in and you’ll find good documentation on them in the Solaris Dynamic Tracing Guide.

The point here is, I hope, obvious. While embedded static probes in code is nice and definitely convenient, its by no means whatsoever a necessity. Scripts like the simple one above will amaze your friends and even if you don’t care to learn DTrace yourself, run it on a process that you think is idle… you’ll be surprised how many apps that “aren’t doing anything” in fact are.

Happy tracing!