Explore Your Storage with FileBench
Posted on July 18, 2008
FileBench is one of the most powerful and flexible benchmarking tools around. Your typical tool like Bonnie++ or IOzone tend to take some discrete operation and do it multiple times at differing block sizes (8K file in 1K blocks, 8K file in 2K blocks, etc). These commonly used benchmarks are known as “micro-benchmarks”. Using them tends to be controversial and can be confusing, leading to claims like “My new Seagate disk gets 800MB/s!!!” In order to make them useful you need to use DirectIO or preform operations that are larger than you installed RAM to avoid “cache effect”.
By contrast, FileBench is better described as an “application simulator” or as I prefer to call it, a “workload generator”. Whereas most benchmarks may use only a single file, FileBench creates filesets prior to actually running a workload. In this way, it can pre-create thousands of files in hundreds of directories with file of varying sizes (all with in given ranges) on which to actually test. This gives you much more realistic ideas of what performance may actually look like in production.
FileBench workloads are actually scripts in the “F” language which define flowops, such as createfile, deletefile, fsync, closefile, etc. This means that you can effectively model bazaar or unusual scenarios, like creating thousands of new files, writing 1 byte, and closing each. Furthermore, because we’re working on a much larger scale, we can leave caching enabled to see how caching helps or hurts a workload.
All that said, FileBench is somewhat non-intuative and is in a lot of flux now. These things are slowly being worked out, and I myself am trying to chip in, but there is a lot of polish to be slapped on this thing.
To get started with FileBench, download the latest version. Packages are available for Solaris (X86 & SPARC), as well as source. Please note that if you install on SPARC or in a Zone the “isaexec” links will fail… in that case, just copy the appropriate binaries (amd64/ or sparc/) into /opt/filebench/bin. Furthermore, please note that the “amd64/” is a farce, FileBench is distributed 32bit X86, not 64bit. (That’ll be fixed soon.)
Running a Single Workload
In /opt/filebench/bin/(platform/) you will find a binary called “go_filebench”, this is Filebench itself. Invoking it will start an interactive shell. Here you can load a workload to run. The workloads are found in /opt/filebench/workloads. The workloads are “F” script files that you can look at and modify to fit your specific need. Each workload has variables associated with it that determine where to run the benchmark (the directory or filesystem you wish to test), number of files, filesize, thread count, etc. We can either create a custom workload with the variable values we want, or we can modify them in the interactive shell prior to run.
Lets take a simple example. In this case I’ve decided to run the “varmail” workload on “/pool/test” (a ZFS dataset):
root@ultra ~$ /opt/filebench/bin/amd64/go_filebench FileBench Version 1.3.3 filebench> load varmail 8429: 4.475: Varmail Version 2.1 personality successfully loaded 8429: 4.475: Usage: set $dir=8429: 4.475: set $filesize= defaults to 16384 8429: 4.476: set $nfiles= defaults to 1000 8429: 4.476: set $nthreads= defaults to 16 8429: 4.476: set $meaniosize= defaults to 16384 8429: 4.476: set $readiosize= defaults to 1048576 8429: 4.476: set $meandirwidth= defaults to 1000000 8429: 4.476: (sets mean dir width and dir depth is calculated as log (width, nfiles) 8429: 4.476: dirdepth therefore defaults to dir depth of 1 as in postmark 8429: 4.476: set $meandir lower to increase depth beyond 1 if desired) 8429: 4.476: 8429: 4.476: run runtime (e.g. run 60) filebench> set $dir=/pool/test filebench> run 60 8429: 39.650: Creating/pre-allocating files and filesets 8429: 39.656: Fileset bigfileset: 1000 files, avg dir = 1000000, avg depth = 0.5, mbyte s=15 8429: 39.657: Creating fileset bigfileset... 8429: 46.876: Preallocated 812 of 1000 of fileset bigfileset in 8 seconds 8429: 46.876: waiting for fileset pre-allocation to finish 8429: 46.876: Starting 1 filereader instances 8430: 47.883: Starting 16 filereaderthread threads 8429: 50.893: Running... 8429: 111.443: Run took 60 seconds... 8429: 111.445: Per-Operation Breakdown closefile4 382ops/s 0.0mb/s 0.0ms/op 5us/op-cpu readfile4 382ops/s 6.3mb/s 0.0ms/op 28us/op-cpu openfile4 382ops/s 0.0mb/s 0.0ms/op 29us/op-cpu closefile3 382ops/s 0.0mb/s 0.0ms/op 6us/op-cpu fsyncfile3 382ops/s 0.0mb/s 19.8ms/op 31us/op-cpu appendfilerand3 382ops/s 3.0mb/s 0.0ms/op 43us/op-cpu readfile3 382ops/s 6.3mb/s 0.0ms/op 28us/op-cpu openfile3 382ops/s 0.0mb/s 0.0ms/op 29us/op-cpu closefile2 382ops/s 0.0mb/s 0.0ms/op 6us/op-cpu fsyncfile2 382ops/s 0.0mb/s 20.8ms/op 34us/op-cpu appendfilerand2 382ops/s 3.0mb/s 0.0ms/op 32us/op-cpu createfile2 382ops/s 0.0mb/s 0.1ms/op 71us/op-cpu deletefile1 382ops/s 0.0mb/s 0.0ms/op 44us/op-cpu 8429: 111.445: IO Summary: 300414 ops 4961.5 ops/s, (763/763 r/w) 18.6mb/s, 145us cpu/op, 10.2ms latency 8429: 111.445: Shutting down processes filebench> quit
Here I loaded the “varmail” workload (actual file is “/opt/filebench/workloads/varmail.f”) and set the directory to run in as my test directory. The rest of the defaults I leave alone. In the output we see that first it created a fileset, in this case it created 1000 files in 1 directory, each file with a random size…. here’s a look:
root@ultra 00000001$ ls -lh | more total 11M -rw-r--r-- 1 root root 3.7K Jul 18 15:31 00000001 -rw-r--r-- 1 root root 7.6K Jul 18 15:31 00000002 -rw-r--r-- 1 root root 11K Jul 18 15:31 00000003 -rw-r--r-- 1 root root 5.6K Jul 18 15:31 00000004 -rw-r--r-- 1 root root 13K Jul 18 15:31 00000005 -rw-r--r-- 1 root root 1.4K Jul 18 15:31 00000006 -rw-r--r-- 1 root root 16K Jul 18 15:31 00000007 -rw-r--r-- 1 root root 921 Jul 18 15:31 00000008 -rw-r--r-- 1 root root 8.2K Jul 18 15:31 00000009 -rw-r--r-- 1 root root 15K Jul 18 15:31 00000010 -rw-r--r-- 1 root root 11K Jul 18 15:31 00000011 -rw-r--r-- 1 root root 7.2K Jul 18 15:31 00000012 -rw-r--r-- 1 root root 7.9K Jul 18 15:31 00000013 -rw-r--r-- 1 root root 2.7K Jul 18 15:31 00000014 -rw-r--r-- 1 root root 1.1K Jul 18 15:31 00000015
By tweeking the variables we can increase the spread.
Finally, in the output we see that the workload ran for the time (in seconds) that we specified and then dumped out both op specific and aggregate stats. For instance, we can easily see in the output that its fsync’s that are the most time consuming operation.
Running Multiple Workloads with BenchPoint
Instead of running just a single workload we commonly want to run several. This might be different workloads or even the same workload repeatedly but with different settings. We can do that with BenchPoint (currently named: “/opt/filebench/bin/filebench”; confusing I know.)
BenchPoint is actually a PERL framework around the FileBench (“go_filebench”) binary. It utilizes a profile that defines all the workloads we with to run as well as all the variable settings for those. It then uses some additional scripts to handle special operations (such as exporting/importing ZFS pools after each run) and stats collection (such as watching vmstat during a run).
To get started, go into the /opt/filebench/config/ directory and look at the various *.prof files. When you see one you like, copy it to some other location, such as /tmp. Do not use the profiles in config/ as is!!! Customize them somewhere else!!! Now edit to taste, changing the global $dir to the location you wish to execute the workloads, etc.
Now that we have a customized profile, lets give it a run:
root@ultra config$ cp filemicro.prof /tmp/benr_filemicro.prof
root@ultra config$ cd /tmp
root@ultra tmp$ vi benr_filemicro.prof
...
root@ultra tmp$ more benr_filemicro.prof
# ident "@(#)filemicro.prof 1.2 08/03/31 SMI"
DEFAULTS {
runtime = 60;
dir = /pool/test/;
stats = /tmp/stats;
filesystem = nofs;
description = "FileMicro Testing";
}
Please note, I’m using ZFS but I specified the above filesystem as “nofs”… thats because if you specify “zfs” benchpoint will export/import the zpool prior to each workload… if you do not want this, specify anything other than “zfs”.
Now lets run this profile. Please note, you need to be in the local directory with your custom profile and you must omit the “.prof” suffix.
root@ultra tmp$ /opt/filebench/bin/filebench benr_filemicro parsing profile for config: createandalloc Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/thisrun.f FileBench Version 1.3.3 8458: 0.021: FileMicro-Create Version 2.1 personality successfully loaded 8458: 0.021: Creating/pre-allocating files and filesets 8458: 0.021: File largefile: mbytes=0 8458: 0.021: Creating file largefile... 8458: 0.021: Preallocated 1 of 1 of file largefile in 1 seconds 8458: 0.021: waiting for fileset pre-allocation to finish 8458: 0.022: Running '/opt/filebench/scripts/fs_flush nofs /pool/test/' filesystem type is: nofs, no action required, so exiting 8458: 0.031: Change dir to /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc 8458: 0.031: Starting 1 filecreater instances 8461: 1.035: Starting 1 filecreaterthread threads 8458: 4.045: Running... 8458: 5.055: Run took 1 seconds... 8458: 5.055: Per-Operation Breakdown finish 507ops/s 0.0mb/s 0.0ms/op 2us/op-cpu append-file 508ops/s 507.0mb/s 1.6ms/op 1615us/op-cpu 8458: 5.055: IO Summary: 513 ops 508.0 ops/s, (0/508 r/w) 507.0mb/s, 1666us cpu/op, 1.6ms latency 8458: 5.055: Stats dump to file 'stats.createandalloc.out' 8458: 5.055: in statsdump stats.createandalloc.out 8458: 5.055: Shutting down processes Generating html for /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s file = /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/stats.createandalloc.out parsing profile for config: createandallocsync Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandallocsync/thisrun.f FileBench Version 1.3.3 8469: 0.012: FileMicro-Create Version 2.1 personality successfully loaded ....
The key to success with FileBench isn’t in the stats that it outputs, but rather by stats you can gather during its load. Use tools like zpool iostat, iostat, vmstat, or even DTrace.
FileBench also includes several handy utilities to assist in your benchmarking, but I’ll discuss those separately in the future.
For more info on FileBench try these links: