If you’ve seen one of my talks on ZFS Tuning you’ve witnessed my love for the fsinfo provider (and the fsstat command as well) first hand. This amazing but poorly documented provider can give you some great high level insight into your I/O pattern. If you’ve ever used truss just to watch iops, then give this a whirl instead.

There are probes for each VFS operation:

   ID   PROVIDER            MODULE                          FUNCTION NAME
14937     fsinfo           genunix                       fop_vnevent vnevent
14938     fsinfo           genunix                       fop_shrlock shrlock
14939     fsinfo           genunix                    fop_getsecattr getsecattr
14940     fsinfo           genunix                    fop_setsecattr setsecattr
14941     fsinfo           genunix                       fop_dispose dispose
14942     fsinfo           genunix                       fop_dumpctl dumpctl
14943     fsinfo           genunix                        fop_pageio pageio
14944     fsinfo           genunix                      fop_pathconf pathconf
14945     fsinfo           genunix                          fop_dump dump
14946     fsinfo           genunix                          fop_poll poll
14947     fsinfo           genunix                        fop_delmap delmap
14948     fsinfo           genunix                        fop_addmap addmap
14949     fsinfo           genunix                           fop_map map
14950     fsinfo           genunix                       fop_putpage putpage
14951     fsinfo           genunix                       fop_getpage getpage
14952     fsinfo           genunix                        fop_realvp realvp
14953     fsinfo           genunix                         fop_space space
14954     fsinfo           genunix                        fop_frlock frlock
14955     fsinfo           genunix                           fop_cmp cmp
14956     fsinfo           genunix                          fop_seek seek
14957     fsinfo           genunix                      fop_rwunlock rwunlock
14958     fsinfo           genunix                        fop_rwlock rwlock
14959     fsinfo           genunix                           fop_fid fid
14960     fsinfo           genunix                      fop_inactive inactive
14961     fsinfo           genunix                         fop_fsync fsync
14962     fsinfo           genunix                      fop_readlink readlink
14963     fsinfo           genunix                       fop_symlink symlink
14964     fsinfo           genunix                       fop_readdir readdir
14965     fsinfo           genunix                         fop_rmdir rmdir
14966     fsinfo           genunix                         fop_mkdir mkdir
14967     fsinfo           genunix                        fop_rename rename
14968     fsinfo           genunix                          fop_link link
14969     fsinfo           genunix                        fop_remove remove
14970     fsinfo           genunix                        fop_create create
14971     fsinfo           genunix                        fop_lookup lookup
14972     fsinfo           genunix                        fop_access access
14973     fsinfo           genunix                       fop_setattr setattr
14974     fsinfo           genunix                       fop_getattr getattr
14975     fsinfo           genunix                         fop_setfl setfl
14976     fsinfo           genunix                         fop_ioctl ioctl
14977     fsinfo           genunix                         fop_write write
14978     fsinfo           genunix                          fop_read read
14979     fsinfo           genunix                         fop_close close
14980     fsinfo           genunix                          fop_open open

The arguments to the probes are:
• args[0]: fileinfo_t *
• args[1]: Return value (0 for success, in the case of writes this is the write size in bytes (ssize_t)).

So lets look at the most generic script we could write:

#pragma D option quiet

fsinfo:genunix:: 
{ 
printf("%s (%s) %s: %d [%s]n", probename, execname, args[0]->fi_pathname, args[1], args[0]->fi_fs);
}

The output specifies the probename (operation), process name that generated the op, the pathname if applicable, the return value, and finally the filesystem type:

$ dtrace -s fsinfo.d 
lookup (idmapd) /etc: 0 [ufs]
lookup (idmapd) /etc/resolv.conf: 0 [ufs]
getattr (idmapd) /etc/resolv.conf: 0 [ufs]
poll (idmapd) <unknown>: 0 [sockfs]
poll (idmapd) <unknown>: 0 [sockfs]
poll (idmapd) <unknown>: 0 [sockfs]
...
close (mysqld) /local/zone/root/var/tmp/#sql_47d3_0.MYI: 0 [zfs]
close (mysqld) /local/zone/root/var/tmp/#sql_47d3_0.MYD: 0 [zfs]
lookup (mysqld) /local/zone/root/var: 0 [zfs]
lookup (mysqld) /local/zone/root/var/tmp: 0 [zfs]
...

This can quickly be jazzed up by adding a conditional by zonename or filesystem type or process. We could do some aggregations rather than the play-by-play.

While the fsstat command can give you per-operation counts by mount, this gives you a lot more data which in turn can be used to find unusual I/O’s that you didn’t expect or help you construct realistic benchmarks using tools like FileBench.