FileBench is one of the most powerful and flexible benchmarking tools around. Your typical tool like Bonnie++ or IOzone tend to take some discrete operation and do it multiple times at differing block sizes (8K file in 1K blocks, 8K file in 2K blocks, etc). These commonly used benchmarks are known as “micro-benchmarks”. Using them tends to be controversial and can be confusing, leading to claims like “My new Seagate disk gets 800MB/s!!!” In order to make them useful you need to use DirectIO or preform operations that are larger than you installed RAM to avoid “cache effect”.

By contrast, FileBench is better described as an “application simulator” or as I prefer to call it, a “workload generator”. Whereas most benchmarks may use only a single file, FileBench creates filesets prior to actually running a workload. In this way, it can pre-create thousands of files in hundreds of directories with file of varying sizes (all with in given ranges) on which to actually test. This gives you much more realistic ideas of what performance may actually look like in production.

FileBench workloads are actually scripts in the “F” language which define flowops, such as createfile, deletefile, fsync, closefile, etc. This means that you can effectively model bazaar or unusual scenarios, like creating thousands of new files, writing 1 byte, and closing each. Furthermore, because we’re working on a much larger scale, we can leave caching enabled to see how caching helps or hurts a workload.

All that said, FileBench is somewhat non-intuative and is in a lot of flux now. These things are slowly being worked out, and I myself am trying to chip in, but there is a lot of polish to be slapped on this thing.

To get started with FileBench, download the latest version. Packages are available for Solaris (X86 & SPARC), as well as source. Please note that if you install on SPARC or in a Zone the “isaexec” links will fail… in that case, just copy the appropriate binaries (amd64/ or sparc/) into /opt/filebench/bin. Furthermore, please note that the “amd64/” is a farce, FileBench is distributed 32bit X86, not 64bit. (That’ll be fixed soon.)

Running a Single Workload

In /opt/filebench/bin/(platform/) you will find a binary called “go_filebench”, this is Filebench itself. Invoking it will start an interactive shell. Here you can load a workload to run. The workloads are found in /opt/filebench/workloads. The workloads are “F” script files that you can look at and modify to fit your specific need. Each workload has variables associated with it that determine where to run the benchmark (the directory or filesystem you wish to test), number of files, filesize, thread count, etc. We can either create a custom workload with the variable values we want, or we can modify them in the interactive shell prior to run.

Lets take a simple example. In this case I’ve decided to run the “varmail” workload on “/pool/test” (a ZFS dataset):

root@ultra ~$ /opt/filebench/bin/amd64/go_filebench
FileBench Version 1.3.3
filebench> load varmail
 8429: 4.475: Varmail Version 2.1 personality successfully loaded
 8429: 4.475: Usage: set $dir=
 8429: 4.475:        set $filesize=    defaults to 16384
 8429: 4.476:        set $nfiles=     defaults to 1000
 8429: 4.476:        set $nthreads=   defaults to 16
 8429: 4.476:        set $meaniosize= defaults to 16384
 8429: 4.476:        set $readiosize=  defaults to 1048576
 8429: 4.476:        set $meandirwidth= defaults to 1000000
 8429: 4.476: (sets mean dir width and dir depth is calculated as log (width, nfiles)
 8429: 4.476:  dirdepth therefore defaults to dir depth of 1 as in postmark
 8429: 4.476:  set $meandir lower to increase depth beyond 1 if desired)
 8429: 4.476:
 8429: 4.476:        run runtime (e.g. run 60)
filebench> set $dir=/pool/test
filebench> run 60
 8429: 39.650: Creating/pre-allocating files and filesets
 8429: 39.656: Fileset bigfileset: 1000 files, avg dir = 1000000, avg depth = 0.5, mbyte
s=15
 8429: 39.657: Creating fileset bigfileset...
 8429: 46.876: Preallocated 812 of 1000 of fileset bigfileset in 8 seconds
 8429: 46.876: waiting for fileset pre-allocation to finish
 8429: 46.876: Starting 1 filereader instances
 8430: 47.883: Starting 16 filereaderthread threads
 8429: 50.893: Running...
 8429: 111.443: Run took 60 seconds...
 8429: 111.445: Per-Operation Breakdown
closefile4                382ops/s   0.0mb/s      0.0ms/op        5us/op-cpu
readfile4                 382ops/s   6.3mb/s      0.0ms/op       28us/op-cpu
openfile4                 382ops/s   0.0mb/s      0.0ms/op       29us/op-cpu
closefile3                382ops/s   0.0mb/s      0.0ms/op        6us/op-cpu
fsyncfile3                382ops/s   0.0mb/s     19.8ms/op       31us/op-cpu
appendfilerand3           382ops/s   3.0mb/s      0.0ms/op       43us/op-cpu
readfile3                 382ops/s   6.3mb/s      0.0ms/op       28us/op-cpu
openfile3                 382ops/s   0.0mb/s      0.0ms/op       29us/op-cpu
closefile2                382ops/s   0.0mb/s      0.0ms/op        6us/op-cpu
fsyncfile2                382ops/s   0.0mb/s     20.8ms/op       34us/op-cpu
appendfilerand2           382ops/s   3.0mb/s      0.0ms/op       32us/op-cpu
createfile2               382ops/s   0.0mb/s      0.1ms/op       71us/op-cpu
deletefile1               382ops/s   0.0mb/s      0.0ms/op       44us/op-cpu

 8429: 111.445:
IO Summary:      300414 ops 4961.5 ops/s, (763/763 r/w)  18.6mb/s,    145us cpu/op,  10.2ms latency
 8429: 111.445: Shutting down processes
filebench> quit

Here I loaded the “varmail” workload (actual file is “/opt/filebench/workloads/varmail.f”) and set the directory to run in as my test directory. The rest of the defaults I leave alone. In the output we see that first it created a fileset, in this case it created 1000 files in 1 directory, each file with a random size…. here’s a look:

root@ultra 00000001$ ls -lh | more
total 11M
-rw-r--r-- 1 root root 3.7K Jul 18 15:31 00000001
-rw-r--r-- 1 root root 7.6K Jul 18 15:31 00000002
-rw-r--r-- 1 root root  11K Jul 18 15:31 00000003
-rw-r--r-- 1 root root 5.6K Jul 18 15:31 00000004
-rw-r--r-- 1 root root  13K Jul 18 15:31 00000005
-rw-r--r-- 1 root root 1.4K Jul 18 15:31 00000006
-rw-r--r-- 1 root root  16K Jul 18 15:31 00000007
-rw-r--r-- 1 root root  921 Jul 18 15:31 00000008
-rw-r--r-- 1 root root 8.2K Jul 18 15:31 00000009
-rw-r--r-- 1 root root  15K Jul 18 15:31 00000010
-rw-r--r-- 1 root root  11K Jul 18 15:31 00000011
-rw-r--r-- 1 root root 7.2K Jul 18 15:31 00000012
-rw-r--r-- 1 root root 7.9K Jul 18 15:31 00000013
-rw-r--r-- 1 root root 2.7K Jul 18 15:31 00000014
-rw-r--r-- 1 root root 1.1K Jul 18 15:31 00000015

By tweeking the variables we can increase the spread.

Finally, in the output we see that the workload ran for the time (in seconds) that we specified and then dumped out both op specific and aggregate stats. For instance, we can easily see in the output that its fsync’s that are the most time consuming operation.

Running Multiple Workloads with BenchPoint

Instead of running just a single workload we commonly want to run several. This might be different workloads or even the same workload repeatedly but with different settings. We can do that with BenchPoint (currently named: “/opt/filebench/bin/filebench”; confusing I know.)

BenchPoint is actually a PERL framework around the FileBench (“go_filebench”) binary. It utilizes a profile that defines all the workloads we with to run as well as all the variable settings for those. It then uses some additional scripts to handle special operations (such as exporting/importing ZFS pools after each run) and stats collection (such as watching vmstat during a run).

To get started, go into the /opt/filebench/config/ directory and look at the various *.prof files. When you see one you like, copy it to some other location, such as /tmp. Do not use the profiles in config/ as is!!! Customize them somewhere else!!! Now edit to taste, changing the global $dir to the location you wish to execute the workloads, etc.

Now that we have a customized profile, lets give it a run:

root@ultra config$ cp filemicro.prof /tmp/benr_filemicro.prof
root@ultra config$ cd /tmp
root@ultra tmp$ vi benr_filemicro.prof
...
root@ultra tmp$ more benr_filemicro.prof
# ident "@(#)filemicro.prof     1.2     08/03/31 SMI"

DEFAULTS {
        runtime = 60;
        dir = /pool/test/;
        stats = /tmp/stats;
        filesystem = nofs;
        description = "FileMicro Testing";
}

Please note, I’m using ZFS but I specified the above filesystem as “nofs”… thats because if you specify “zfs” benchpoint will export/import the zpool prior to each workload… if you do not want this, specify anything other than “zfs”.

Now lets run this profile. Please note, you need to be in the local directory with your custom profile and you must omit the “.prof” suffix.

root@ultra tmp$ /opt/filebench/bin/filebench benr_filemicro
parsing profile for config: createandalloc
Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/thisrun.f
FileBench Version 1.3.3
 8458: 0.021: FileMicro-Create Version 2.1 personality successfully loaded
 8458: 0.021: Creating/pre-allocating files and filesets
 8458: 0.021: File largefile: mbytes=0
 8458: 0.021: Creating file largefile...
 8458: 0.021: Preallocated 1 of 1 of file largefile in 1 seconds
 8458: 0.021: waiting for fileset pre-allocation to finish
 8458: 0.022: Running '/opt/filebench/scripts/fs_flush nofs /pool/test/'
filesystem type is: nofs, no action required, so exiting
 8458: 0.031: Change dir to /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc
 8458: 0.031: Starting 1 filecreater instances
 8461: 1.035: Starting 1 filecreaterthread threads
 8458: 4.045: Running...
 8458: 5.055: Run took 1 seconds...
 8458: 5.055: Per-Operation Breakdown
finish                    507ops/s   0.0mb/s      0.0ms/op        2us/op-cpu
append-file               508ops/s 507.0mb/s      1.6ms/op     1615us/op-cpu

 8458: 5.055:
IO Summary:        513 ops 508.0 ops/s, (0/508 r/w) 507.0mb/s,   1666us cpu/op,   1.6ms latency
 8458: 5.055: Stats dump to file 'stats.createandalloc.out'
 8458: 5.055: in statsdump stats.createandalloc.out
 8458: 5.055: Shutting down processes
Generating html for /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s
file = /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/stats.createandalloc.out

parsing profile for config: createandallocsync
Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandallocsync/thisrun.f
FileBench Version 1.3.3
 8469: 0.012: FileMicro-Create Version 2.1 personality successfully loaded
....

The key to success with FileBench isn’t in the stats that it outputs, but rather by stats you can gather during its load. Use tools like zpool iostat, iostat, vmstat, or even DTrace.

FileBench also includes several handy utilities to assist in your benchmarking, but I’ll discuss those separately in the future.

For more info on FileBench try these links:

• FileBench page on Solaris Internals Wiki
• FileBench page on OpenSolaris.org
• Official Home on SourceForge

This entry was posted on Friday, July 18th, 2008 at 8:51 pm and is filed under OpenSolaris. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.