This film has been hugely influential for me, and many others. That awesome unbuttoned shirt over tshirt look that we emulate to this day, decades of trying to get text-to-voice to sound like the film (which was an actor reading the lines word by word in reverse), and who didn't fall in love with Ally Sheedy!!!

Don't be dumb!!! Go! Tonight! 7:30PM!!!

That just sounds like a challenge. Bring it on! Can snoop or tcpdump do this?

root@ultra ~$ ./ip_whosent.d 
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 71 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 72.14.207.99: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 52 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 54 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 87 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 58 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 64 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 65 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 208.67.219.230: 644 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 637 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 72.14.207.99: 660 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 664 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 48 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 72.14.207.99: 40 byte packet on behalf of firefox-bin (PID: 1944)
^C

Here is the script:

#!/usr/sbin/dtrace -qs 



ip:ip:*:send
/execname != "sched"/
{ 
        printf("Packet sent to %s: %d byte packet on behalf of %s (PID: %d)n", 
                        args[2]->ip_daddr, args[4]->ipv4_length, execname, pid ); 
}

Oh but wait....... how about a full call stack on each sent packet? Just add a new line to the above script: stack();

root@ultra ~$ ./ip_sentstack.d 
Packet sent to 72.14.207.99: 84 byte packet on behalf of ping (PID: 2020)

              ip`ip_wput_ire+0x21f5
              ip`ire_send+0x1c9
              ip`ire_add_then_send+0x2b9
              ip`ip_newroute+0xa0a
              ip`ip_output_options+0x18c7
              ip`icmp_wput+0x44a
              unix`putnext+0x22b
              genunix`strput+0x1ad
              genunix`kstrputmsg+0x261
              sockfs`sosend_dgram+0x26e
              sockfs`sotpi_sendmsg+0x4a8
              sockfs`sendit+0x160
              sockfs`sendto+0x8e
              sockfs`sendto32+0x2d
              unix`sys_syscall32+0x101

Or check out one of the examples on the IP Provider wiki page (this is almost certainly by Brendan Gregg):

# ./ipio.d
 CPU  DELTA(us)          SOURCE               DEST      INT  BYTES
   1     598913    10.1.100.123 ->   192.168.10.75  ip.tun0     68
   1         73   192.168.1.108 ->     192.168.5.1     nge0    140
   1      18325   192.168.1.108 <-     192.168.5.1     nge0    140
   1         69    10.1.100.123 <-   192.168.10.75  ip.tun0     68
   0     102921    10.1.100.123 ->   192.168.10.75  ip.tun0     20
   0         79   192.168.1.108 ->     192.168.5.1     nge0     92

Here is the script:

#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %3s %10s %15s    %15s %8s %6sn", "CPU", "DELTA(us)",
            "SOURCE", "DEST", "INT", "BYTES");
        last = timestamp;
}

ip:::send
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s -> %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_saddr, args[2]->ip_daddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

ip:::receive
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s <- %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_daddr, args[2]->ip_saddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

Can DTrace decrypt IPsec ESP payloads? No. Ok, so tcpdump isn't dead yet, but the capabilities offered by DTrace are far deeper. I've got a ton of ideas more that I could put here, but don't have time atm. DTrace for the win!

root@ultra include$ dtrace -qn 'ip:ip:*:receive{ printf("Packet recieved from %s: %d byte packetn", args[2]->ip_saddr, args[4]->ipv4_length ); }'
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 8.11.47.20: 88 byte packet
Packet recieved from 8.11.47.20: 216 byte packet
Packet recieved from 8.11.47.20: 200 byte packet
Packet recieved from 8.11.47.20: 136 byte packet
Packet recieved from 8.11.47.20: 104 byte packet
^C

Pretty soon snoop and tcpdump will be nothing more than unpleasant memories. :)

A big thank you to the DTrace Team!!!

IPsec can be used for direct system-to-system access known as "transport mode" or to create a virtual pipeline into which everything is encrypted, known as "tunnel mode". We're going to look at transport mode, which is an excellent solution for encrypting otherwise unencrypted protocols, such as SNMPv1/2 or telnet.

When encrypting and decrypting data we need keys. This can be done using PKI certificates or IKE generated one-time keys, but in this examples for simplicity sake we'll create our own "static" keys which will be used on both ends of the connection, thus said to be "pre-shared".

Creating Keys

Using the ipsecalgs command we can see the available algorithms, including DES, 3DES, AES, Blowfish, SHA and MD5. Different alogithms require different key lengths, for instance 3DES requires a 192 bit key, whereas Blowfish can use a key anywhere from 32bits up to 448 bits.

For interoperability reasons (such as OSX or Linux), you may with to create keys that are both ASCII and hex. This is done by choosing a string and converting it to hex. To know how long a string should be, divide the number of bits required by 8, this is the number of ASCII chars you need. The hex value of that ASCII string will be double the number of ASCII chars. Using the od utility we can convert ASCII-to-hex. Here I'll create 2 keys, one for AH which is a SHA1 160bit key (20 ASCII chars) and another for ESP which is a Blowfish 256bit key (32 ASCII chars):

benr@ultra ~$ echo "my short ah password" | od -t x1
0000000 6d 79 20 73 68 6f 72 74 20 61 68 20 70 61 73 73
0000020 77 6f 72 64 0a
0000025
benr@ultra ~$ echo "this is my long blowfish esp pas" | od -t x1
0000000 74 68 69 73 20 69 73 20 6d 79 20 6c 6f 6e 67 20
0000020 62 6c 6f 77 66 69 73 68 20 65 73 70 20 70 61 73
0000040 0a
0000041

To ensure proper length, I like using a little text-rule like you see below in vi:

         1         2    2    3 3       4         5         6   6     7  
1234567890 234567890 234567890 234567890 234567890 234567890 234567890
--------------------------------------------------------------------------------------------------------------------
my short ah password
6d792073686f72742061682070617373776f7264

this is my long blowfish esp pas
74686973206973206d79206c6f6e6720626c6f77666973682065737020706173

If you don't require interoperability by knowing the ASCII equivilent, just grab a random set of hex chars (head /dev/random | od -t x1).

Now that we have a key, lets use it.

Configuring IPsec Policies

IPsec policies are rules that the IP stack uses to determine what action should be taken. Actions include:

• bypass: Do nothing, skip the remaining rules if datagram matches.
• drop: Drop if datagram matches.
• permit: Allow if datagram matches, otherwise discard. (Only for inbound datagrams.)
• ipsec: Use IPsec if the datagram matches.

As you can see, this sounds similar to a firewall rule, and to some extent can be used that way, but you ultimately find IPFilter much better suited to that task. When you plan your IPsec environment consider which rules are appropriate in which place.

IPsec policies are defined in the /etc/inet/ipsecinit.conf file, which can be loaded/reloaded using the ipsecconf command. Lets look at a sample configuration:

benr@ultra inet$ cat /etc/inet/ipsecinit.conf 
##
##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ raddr 8.11.80.5 rport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

Our first policy explicitly bypasses connections in and out ("dir both", as in direction) for the local port 22 (SSH). Do I need this here? No, but I include it as an example. You can see the format, the first curly block defines the filter, the second curly block defines parameters, the keyword in between is the action.

The second policy is what we're interested in, its action is ipsec, so if the filter in the first curly block matches we'll use IPsec. "raddr" defines a remote address and "rport" defines a remote port, therefore this policy applies only to outbound connections where we're telnet'ing (port 23) to 8.11.80.5. The second curly block defines parameters for the action, in this case we define the encryption algorithm (Blowfish), encryption authentication algorithm (SHA1), and state that the Security Association is "shared". This is a full ESP connection, meaning we're encrypting and encapsulating the full packet, if we were doing AH (authentication only) we would only define "auth_algs".

Now, on the remote side of the connection (8.11.80.5) we create a similar policy, but rather than "raddr" and "rport" we use "laddr" (local address) and "lport" (local port). We could even go so far as to specify the remote address such that only the specified host would use IPsec to the node. Here's that configuration:

##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ laddr 8.11.80.5 lport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

To load the new policy file you can refresh the ipsec/policy SMF service like so: svcadm refresh ipsec/policy. I recommend avoiding the ipsecconf command except to (without arguments) display the active policy configuration.

So we've defined policies that will encrypt traffic from one node to another, but we're not done yet! We need to define a Security Association that will association keys with our policy.

Creating Security Associations

Security Associations (SAs) can be manually created by either using the ipseckeys command or directly editing the /etc/inet/secret/ipseckeys file, I recommend the latter, I personally find the ipseckeys shell very intimidating.

Lets look at a sample file and then discuss it:

add esp spi 1000 src 8.15.11.17 dst 8.11.80.5 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373
add esp spi 1001 src 8.11.80.5 dst 8.15.11.17 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373

It looks more intimidating that it is. Each line is "add"ing a new static Security Association, both are for ESP. The SPI is the "Security Parameters Index", is a simple numeric value that represents the SA, nothing more, pick any value you like. The src and dst define the addresses to which this SA applies, note that you have two SA's here, one for each direction. Finally, we define the encryption and authentication algorithms and full keys.

I hope that looking at this makes it more clear how policies and SA's fit together. If the IP stack matches a datagram against a policy who's action is "ipsec", it takes the packet and looks for an SA who's address pair matches, and then uses those keys for the action encryption.

Note that if someone obtains your keys your hosed. If you pre-shared keys in this way, change the keys from time-to-time or consider using IKE which can negotiate keys (and thus SAs) on your behalf.

To apply your new SA's, flush and then load using the ipseckeys command:

$ ipseckey flush
$ ipseckey -f /etc/inet/secret/ipseckeys

Is it working? How to Test

All this is for nothing if you don't verify that the packets are actually encrypted. Using snoop, you should see packets like this:

$ snoop -d e1000g0
Using device e1000g0 (promiscuous mode)
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 1 arrived at 9:52:4.58883
ETHER:  Packet size = 90 bytes
ETHER:  Destination = xxxxxxxxxxx, 
ETHER:  Source      = xxxxxxxxxx, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 72 bytes
IP:   Identification = 36989
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 61 seconds/hops
IP:   Protocol = 50 (ESP)
IP:   Header checksum = ab9c
IP:   Source address = XXXXXXXXX
IP:   Destination address = XXXXXXXXXXXX
IP:   No options
IP:   
ESP:  ----- Encapsulating Security Payload -----
ESP:  
ESP:  SPI = 0x3e8
ESP:  Replay = 55
ESP:     ....ENCRYPTED DATA....

And there you go. You can no encrypt communication transparently in the IP stack. Its a little effort to get going, but once its running your done... just remember to rotate those keys every so often!

Why do I care about this again?

In this modern era where SSH is the standard for communication its easy to get jaded. Either you can communicate via SSH or easily create a tunnel to get the job done. But lets face it, SSH is massively overused, and in many cases SSH tunneling is just downright ghetto. With IPsec we can as easily encrypt 100 ports as 1, whereas with SSH thats very ugly. Furthermore, there are many instances in which you want a secure communications channel thats as transparent as possible, such as a network database connection that doesn't offer native encryption or perhaps an SCM or even SNMPv1/2.

While most applications today provide some type of encryption capability it surprising how few people leverage them unless they are the default. In situations where its difficult or impractical to use encryption in the application, IPsec can be a really sweet solution.

IPsec is a technology widely known. Created for IPv6 and backported to IPv4, it adds a security layer into the IP stack. Prior to IPsec we needed to encrypt data before sending it down the stack and then decrypt it on the other side once it came back out using technologies like SSH or SSL/TLS. IPsec simplifies this by transparently handling encrypt/decrypt as well as header authentication.

But for all its intended simplicity and transparency it is to the administrator anything but. If you google around you find piles of email on various lists of those who tried to get IPsec working and gave up in frustration and instead used OpenVPN, on both Linux and Solaris.

Part of the reason IPsec is so complex to manage is the number of technologies that all must work together properly in order to get something functional. This includes acronyms like AH, ESP, IKE, DH, RSA, SA, SAD, etc. In many respects its similar to the frustration encountered the first time you approach LDAP and are overwhelmed with OU=, CN=, DN=, etc. Just like LDAP, with a little practice you get it sorted out and start making ground. We'll start high level and zoom in... please please, read this before just copying examples! (I know you won't, but it'll save you time.)

IPsec is fundamentally similar to a firewall, in that you specify IPsec Policies that determines how datagrams are handled. If a policy matches a datagram its handled... if no policy matches it acts like normal.

Thats where the firewall analogy stops, like a firewall the policy says what to do and when to do it.... but we need to know something else for encrypting or decrypting data, we also need to know how to do it. The information that explains the how is known as a Security Association (SA). This is maintained in a special database named, unremarkably, the Security Association Database (SAD or SADB, depending on who you ask, same thing though).

It's the SA that actually contains the keys used for encrypting, decrypting and authenticating IPsec datagrams. So a packets goes through the stack, a policy says "If the datagram source is 1.2.3.4 and the destination is 5.6.7.8 use IPsec." Now that the stack knows to use IPsec to encrypt, it goes looking in the SADB for an SA that contains the keys. There is a lot of duplication here because they are independent things, so you specify the authentication algorithm, encryption algorithm, source and destination addresses in both the IPsec policy and the SA. Its strange at first. :)

Now, just as IPsec is supposed to make life simple and transparent, so is IKE (Internet Key Exchange). An IKE daemon runs on both sides of a connection and negotiates SA's for you. This provides a variety of benefits, but simplicity isn't exactly one of them because you still need to configure IKE rules which are similar to SA's. In other words, you can't just create an IPsec policy, and then enable IKE and be done.

Before we get into the examples, lets looks at the various files and commands involved:

Files:

    * IPsec:
          o /etc/inet/ipsecinit.conf: IPsec Policy Definitions
          o /etc/inet/secret/ipseckeys: IPsec SA Definitions 
    * IKE:
          o /etc/inet/ike/config: IKE Global Configuation and Rules
          o /etc/inet/secret/ike.preshared: File containing Preshared Key Definitions
          o /etc/inet/secret/ike.privatekeys/: Directory containing IKE Private Keys
          o /etc/inet/ike/publickeys/: Directory containing IKE Public Keys
          o /etc/inet/ike/crls/: Directory containing IKE Certificates 

Commands:

    * IPsec:
          o ipsecconf: Load or Display IPsec Policy Configuration
          o ipsekkey: Manually manipulate IPsec Security Association Database (SADB)
          o ipsecalgs: Display available IPsec ESP/AH Algorithms 
    * IKE:
          o in.iked: IKE Daemon
          o ikeadm: Manipulate IKE parameters and state (flush, add, get, set, ...)
          o ikecert: Manipulate IKE's on-filesystem public-key certificate databases 

The simplest configuration would be to "pre-share" keys, meaning create a key and manually put it on both systems. In my next blog entry we'll step through actually creating an IPsec policy, creating keys, creating Security Associations, and testing an IPsec connection.

By contrast, FileBench is better described as an "application simulator" or as I prefer to call it, a "workload generator". Whereas most benchmarks may use only a single file, FileBench creates filesets prior to actually running a workload. In this way, it can pre-create thousands of files in hundreds of directories with file of varying sizes (all with in given ranges) on which to actually test. This gives you much more realistic ideas of what performance may actually look like in production.

FileBench workloads are actually scripts in the "F" language which define flowops, such as createfile, deletefile, fsync, closefile, etc. This means that you can effectively model bazaar or unusual scenarios, like creating thousands of new files, writing 1 byte, and closing each. Furthermore, because we're working on a much larger scale, we can leave caching enabled to see how caching helps or hurts a workload.

All that said, FileBench is somewhat non-intuative and is in a lot of flux now. These things are slowly being worked out, and I myself am trying to chip in, but there is a lot of polish to be slapped on this thing.

To get started with FileBench, download the latest version. Packages are available for Solaris (X86 & SPARC), as well as source. Please note that if you install on SPARC or in a Zone the "isaexec" links will fail... in that case, just copy the appropriate binaries (amd64/ or sparc/) into /opt/filebench/bin. Furthermore, please note that the "amd64/" is a farce, FileBench is distributed 32bit X86, not 64bit. (That'll be fixed soon.)

Running a Single Workload

In /opt/filebench/bin/(platform/) you will find a binary called "go_filebench", this is Filebench itself. Invoking it will start an interactive shell. Here you can load a workload to run. The workloads are found in /opt/filebench/workloads. The workloads are "F" script files that you can look at and modify to fit your specific need. Each workload has variables associated with it that determine where to run the benchmark (the directory or filesystem you wish to test), number of files, filesize, thread count, etc. We can either create a custom workload with the variable values we want, or we can modify them in the interactive shell prior to run.

Lets take a simple example. In this case I've decided to run the "varmail" workload on "/pool/test" (a ZFS dataset):

root@ultra ~$ /opt/filebench/bin/amd64/go_filebench 
FileBench Version 1.3.3
filebench> load varmail
 8429: 4.475: Varmail Version 2.1 personality successfully loaded
 8429: 4.475: Usage: set $dir=
 8429: 4.475:        set $filesize=    defaults to 16384
 8429: 4.476:        set $nfiles=     defaults to 1000
 8429: 4.476:        set $nthreads=   defaults to 16
 8429: 4.476:        set $meaniosize= defaults to 16384
 8429: 4.476:        set $readiosize=  defaults to 1048576
 8429: 4.476:        set $meandirwidth= defaults to 1000000
 8429: 4.476: (sets mean dir width and dir depth is calculated as log (width, nfiles)
 8429: 4.476:  dirdepth therefore defaults to dir depth of 1 as in postmark
 8429: 4.476:  set $meandir lower to increase depth beyond 1 if desired)
 8429: 4.476:  
 8429: 4.476:        run runtime (e.g. run 60)
filebench> set $dir=/pool/test
filebench> run 60
 8429: 39.650: Creating/pre-allocating files and filesets
 8429: 39.656: Fileset bigfileset: 1000 files, avg dir = 1000000, avg depth = 0.5, mbyte                  
s=15
 8429: 39.657: Creating fileset bigfileset...
 8429: 46.876: Preallocated 812 of 1000 of fileset bigfileset in 8 seconds
 8429: 46.876: waiting for fileset pre-allocation to finish
 8429: 46.876: Starting 1 filereader instances
 8430: 47.883: Starting 16 filereaderthread threads
 8429: 50.893: Running...
 8429: 111.443: Run took 60 seconds...
 8429: 111.445: Per-Operation Breakdown
closefile4                382ops/s   0.0mb/s      0.0ms/op        5us/op-cpu
readfile4                 382ops/s   6.3mb/s      0.0ms/op       28us/op-cpu
openfile4                 382ops/s   0.0mb/s      0.0ms/op       29us/op-cpu
closefile3                382ops/s   0.0mb/s      0.0ms/op        6us/op-cpu
fsyncfile3                382ops/s   0.0mb/s     19.8ms/op       31us/op-cpu
appendfilerand3           382ops/s   3.0mb/s      0.0ms/op       43us/op-cpu
readfile3                 382ops/s   6.3mb/s      0.0ms/op       28us/op-cpu
openfile3                 382ops/s   0.0mb/s      0.0ms/op       29us/op-cpu
closefile2                382ops/s   0.0mb/s      0.0ms/op        6us/op-cpu
fsyncfile2                382ops/s   0.0mb/s     20.8ms/op       34us/op-cpu
appendfilerand2           382ops/s   3.0mb/s      0.0ms/op       32us/op-cpu
createfile2               382ops/s   0.0mb/s      0.1ms/op       71us/op-cpu
deletefile1               382ops/s   0.0mb/s      0.0ms/op       44us/op-cpu

 8429: 111.445: 
IO Summary:      300414 ops 4961.5 ops/s, (763/763 r/w)  18.6mb/s,    145us cpu/op,  10.2ms latency
 8429: 111.445: Shutting down processes
filebench> quit

Here I loaded the "varmail" workload (actual file is "/opt/filebench/workloads/varmail.f") and set the directory to run in as my test directory. The rest of the defaults I leave alone. In the output we see that first it created a fileset, in this case it created 1000 files in 1 directory, each file with a random size.... here's a look:

root@ultra 00000001$ ls -lh | more
total 11M
-rw-r--r-- 1 root root 3.7K Jul 18 15:31 00000001
-rw-r--r-- 1 root root 7.6K Jul 18 15:31 00000002
-rw-r--r-- 1 root root  11K Jul 18 15:31 00000003
-rw-r--r-- 1 root root 5.6K Jul 18 15:31 00000004
-rw-r--r-- 1 root root  13K Jul 18 15:31 00000005
-rw-r--r-- 1 root root 1.4K Jul 18 15:31 00000006
-rw-r--r-- 1 root root  16K Jul 18 15:31 00000007
-rw-r--r-- 1 root root  921 Jul 18 15:31 00000008
-rw-r--r-- 1 root root 8.2K Jul 18 15:31 00000009
-rw-r--r-- 1 root root  15K Jul 18 15:31 00000010
-rw-r--r-- 1 root root  11K Jul 18 15:31 00000011
-rw-r--r-- 1 root root 7.2K Jul 18 15:31 00000012
-rw-r--r-- 1 root root 7.9K Jul 18 15:31 00000013
-rw-r--r-- 1 root root 2.7K Jul 18 15:31 00000014
-rw-r--r-- 1 root root 1.1K Jul 18 15:31 00000015

By tweeking the variables we can increase the spread.

Finally, in the output we see that the workload ran for the time (in seconds) that we specified and then dumped out both op specific and aggregate stats. For instance, we can easily see in the output that its fsync's that are the most time consuming operation.

Running Multiple Workloads with BenchPoint

Instead of running just a single workload we commonly want to run several. This might be different workloads or even the same workload repeatedly but with different settings. We can do that with BenchPoint (currently named: "/opt/filebench/bin/filebench"; confusing I know.)

BenchPoint is actually a PERL framework around the FileBench ("go_filebench") binary. It utilizes a profile that defines all the workloads we with to run as well as all the variable settings for those. It then uses some additional scripts to handle special operations (such as exporting/importing ZFS pools after each run) and stats collection (such as watching vmstat during a run).

To get started, go into the /opt/filebench/config/ directory and look at the various *.prof files. When you see one you like, copy it to some other location, such as /tmp. Do not use the profiles in config/ as is!!! Customize them somewhere else!!! Now edit to taste, changing the global $dir to the location you wish to execute the workloads, etc.

Now that we have a customized profile, lets give it a run:

root@ultra config$ cp filemicro.prof /tmp/benr_filemicro.prof
root@ultra config$ cd /tmp
root@ultra tmp$ vi benr_filemicro.prof 
...
root@ultra tmp$ more benr_filemicro.prof 
# ident "@(#)filemicro.prof     1.2     08/03/31 SMI"

DEFAULTS {
        runtime = 60;
        dir = /pool/test/;
        stats = /tmp/stats;
        filesystem = nofs;
        description = "FileMicro Testing";
}

Please note, I'm using ZFS but I specified the above filesystem as "nofs"... thats because if you specify "zfs" benchpoint will export/import the zpool prior to each workload... if you do not want this, specify anything other than "zfs".

Now lets run this profile. Please note, you need to be in the local directory with your custom profile and you must omit the ".prof" suffix.

root@ultra tmp$ /opt/filebench/bin/filebench benr_filemicro      
parsing profile for config: createandalloc
Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/thisrun.f
FileBench Version 1.3.3
 8458: 0.021: FileMicro-Create Version 2.1 personality successfully loaded
 8458: 0.021: Creating/pre-allocating files and filesets
 8458: 0.021: File largefile: mbytes=0
 8458: 0.021: Creating file largefile...
 8458: 0.021: Preallocated 1 of 1 of file largefile in 1 seconds
 8458: 0.021: waiting for fileset pre-allocation to finish
 8458: 0.022: Running '/opt/filebench/scripts/fs_flush nofs /pool/test/'
filesystem type is: nofs, no action required, so exiting
 8458: 0.031: Change dir to /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc
 8458: 0.031: Starting 1 filecreater instances
 8461: 1.035: Starting 1 filecreaterthread threads
 8458: 4.045: Running...
 8458: 5.055: Run took 1 seconds...
 8458: 5.055: Per-Operation Breakdown
finish                    507ops/s   0.0mb/s      0.0ms/op        2us/op-cpu
append-file               508ops/s 507.0mb/s      1.6ms/op     1615us/op-cpu

 8458: 5.055: 
IO Summary:        513 ops 508.0 ops/s, (0/508 r/w) 507.0mb/s,   1666us cpu/op,   1.6ms latency
 8458: 5.055: Stats dump to file 'stats.createandalloc.out'
 8458: 5.055: in statsdump stats.createandalloc.out
 8458: 5.055: Shutting down processes
Generating html for /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s
file = /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandalloc/stats.createandalloc.out

parsing profile for config: createandallocsync
Running /tmp/stats/ultra-nofs-benr_filemicro-Jul_18_2008-15h_50m_11s/createandallocsync/thisrun.f
FileBench Version 1.3.3
 8469: 0.012: FileMicro-Create Version 2.1 personality successfully loaded
....

The key to success with FileBench isn't in the stats that it outputs, but rather by stats you can gather during its load. Use tools like zpool iostat, iostat, vmstat, or even DTrace.

FileBench also includes several handy utilities to assist in your benchmarking, but I'll discuss those separately in the future.

For more info on FileBench try these links:

• FileBench page on Solaris Internals Wiki
• FileBench page on OpenSolaris.org
• Official Home on SourceForge

When I think about what Sun concept of Open Storage really boils down to it is this: servers aren't just storage clients. If you think about the direction Fibre Channel and even iSCSI solutions were going, the drive was to push more and more of the storage management and access into array controllers such that servers are clients only. I think I told the story in this blog some time ago when I stormed out of HDS's data center when I realized you required a Windows server to manage the array. Storage should be autonomous!

But things have changed. When I stormed out of HDS I was managing an environment of large SPARC systems that had 1 or 2 internal disks just for the OS, or small 1U X86 servers with just enough local disk for the OS and apps. With the increasing availability of high performance multi-core CPU's 2U's are more attractive and local disk storage is commonly managed by a dedicated RAID card with onboard cache of up to 512MB. When you have racks full of 2U systems that each have more than 2.5TB of RAID6 and a write-back cache to boot in each machine... its time to think differently. Filesystems like Lustre or even pNFS (parallel NFS) look very attractive to the enterprise.... yet again, HPC technology trickles down to the enterprise market.

While the push from Sun has just started publicly this year, there has been signs of this for a long time, especially when Jonathan declared many moons ago that all proprietary OS's would have to go, which at the time was shocking given that all the storage arrays ran various embedded or specialized OS's. So, it should be noted that this would seem to be the fulfillment of something Sun has been working toward for quite some time, unified under a single banner of "Open Storage".

The implications could really change the landscape though. Traditionally in large enterprise storage you spend a lot of time working with vendors, testing configurations, listening to presos, etc. It was a very hands-off world. This new push would mean that Storage Administrators are going to spend less time making purchasing decisions and more time learning how to install, manage, and optimize their deployments. When "secure storage" goes from checking a box to configuring IPsec things get sticky. But that also provides new opportunities for administrators and vendors alike. In fact, that reminds me of something.... :)

So the real question is, how will "traditional" storage vendors like HDS and EMC respond? If you don't have a server business getting behind the idea of buying servers and JBOD's isn't terribly attractive. That suggests that in 3 years companies like Dell, Sun, IBM, and HP will rule the storage world leaving EMC to supply a dying market while it continues to cash in on its acquisitions like VMware and RSA.

So, like I said in the beginning.... "Open Storage" is either something mind-numbingly obvious or something radically new, depending on where you sit.

So, there are 3 arrays, we'll look at them each.

The Sun Storage J4200 array is an single or dual controller external SAS JBOD and offers 3 SAS ports per controller. It supports "Hardware Raid", but notice that its "(with RAID HBA)", so there is no hardware RAID happening on the controllers (the same goes for similar solutions from Dell). This unit is 2U and features 12 3.5" drives. While the interconnect is SAS, you can use either SAS or SATA drives. The cheapest setup is single controller with 2 250GB disks, for $3,140.00. If you customize one with a fairly normal config of dual controllers, 12x 500GB SATA drives (7200RPM), plus 1 cable, 1 HBA, and a rail kit you come up to around $9,000 with a raw capacity of 6TB.

The Sun Storage J4400 Array is the same basic array but with more disk capacity. You still don't get hardware RAID in the controllers, only on the "RAID HBA". The chassis accommodates up to 24 drives in 4U, and starts at $7,410.00, although that price is single controller with 12x 73GB SAS drives. A reasonably stocked config with 24x 500GB SATA drives, dual controllers, dual HBA's, cables, and rack kit come to just over $20,000 with a raw capacity of 12TB.

Lastly, the one many people have been waiting for... the Sun Storage J4500 Array. This is a Thumper but notice whats wrong in the picture above? The server component is replaced with SAS controllers. This is a 48 disk in 4U storage array in the traditional sense, not a hybrid server. The unit does not feature independently replaceable controllers, but otherwise shares in the basic vibe of its siblings in the lineup. "Four 3 Gb/sec SAS ports per tray", however 2 are server ports and 2 are expansion (daisy chain) ports. Prices start at $32,960.00 for a 48x 500GB SATA setup (24TB raw), and go up to $60,960.00 for a 24x 1TB SATA setup (48TB raw). SAS disks are not an option in the unit, and you will have to use a "RAID HBA" for "hardware RAID".

One thing I'll note is that they are branded "Sun Storage", rather than "StorageTek". I did find one or two "Sun StorageTek J4000" references around but very few and they looked like mistakes. I'm supportive. :)

In general, I think its good to see Sun pushing new storage product. Does it differentiate enough from the offerings in its line? I dunno. Clearly Sun is addressing the demise of Fibre Channel is a large segment of the market, but competition in that space is with Dell and the like and very competitive. For instance, a Dell MD1000 configured like the J4200 above (12x 500GB SATA, dual controller standard, single HBA, cable, rack kit, etc) is $7,000... versus $9,000 for a J4200 with only a single controller. Which is better? That comes down to the HBA actually, and I'm terrified (RAID 1E? I only see that on Adaptec) the HBA is an Adaptec controller rebranded as StorageTek. The Dell PERC HBA's (LSI MegaSAS) are the best around, hands down.

What Sun has that Dell, or anyone else, doesn't have is the software. Lustre is now Sun, ZFS is Sun, SAMFS/QFS are Sun, not to mention HoneyComb and Sun's work on pNFS. Sun has the storage software unlike any other vendor in the industry. I can only hope that Sun is pushing the "Open Storage" to raise awareness, not of something new, but what its been dedicated to for some time as part of OpenSolaris. If that awareness rises and low barrier to entry gets customers excited they may, hopefully, be willing to pay more money because they'll have support for hardware and software from a single vendor. Lets hope. ;)

Every so often Penny Arcade does a Jesus comic... I commonly find myself being amused and offended at the same time, and thats always a winning combo. Here's my favorite example:

Whats funny is how much theology is in there. Namely, "We just..." expresses trinitarian belief. (God is 1 God in 3 Persons: God the Father, God the Son, God the Holy Spirit... in Genesis Chapter 1 you can see reference to all 3, and any debate should be handled by looking at Gen 1:26 "Then God said, “Let us make man[8] in our image, after our likeness."... fascinating topic of study.)

The other thing I like about the strip is that it has the right attitude. We are told that "Fear of the Lord is the beginning of wisdom"... which stands to reason, if you want to know wisdom understand the creator of all things and realize His power and authority. Yet we're also called into a personal relationship with Jesus, our savior, and the indwelling Holy Spirit. People get too legalistic about Christianity... you don't leave God in the church on sunday, you're constantly in relationship to Him, even if your ignoring Him.

One evidence of our relationship with Jesus, whether we believe or not, is our .conscience There are universal right and wrongs regardless of culture. Sure, you can numb yourself to it or be taught otherwise, but there are instinctual "thats wrong" feelings in our hearts. The Bible tells us that the law of God is inscribed in our hearts. Before you listen to a preacher, look at your own heart and what it tells you and see if the Bible lines up with that.

I don't believe in Jesus Christ as my savior because I want to go to heaven or because it makes me a good person or something... I believe because when I pair up whats in my heart with all the explanations and writings available the Bible is the only book that jives with reality. There is unity and organization both physically and philosophically all throughout nature, common themes, which is why we can explain complex philosophical concepts by illustration in nature. A single intellectual creator explains that. Many religions tell me what I can become, but Jesus describes what I am, why I am that way, and how I can change it if I choose. And Jesus gives me a choice and clearly outlines the consequences of either decision. The Bible outlines how God has interacted with man in different ways over time and why they didn't work... "Just give me a sign and I'll believe!", and He did that and it didn't make a difference, "Just give me some easy rules to follow and I'll believe!", and He did that and it didn't make a difference. It all leads up to Jesus and grace. We don't deserve crap, but He created us and loves us all, no matter what... we are present in life with a choice, Him or not Him. Simple, beautiful, wonderful. Thats something I can get behind.

I've always been discourage in the modern American church, which is dominated by old, legalistic, men. But Jesus was 30 when we started his mission... the apostles were all young as well. So why are young men relegated to youth pastors in the modern church? I think its a big reason that Christianity has such a bad rap in America and churches are dying left and right. I'm young, I'm aggressive, I love Jesus... so was Peter! When they came to take Jesus Peter busts out his sword and starts chopping! It was the wrong thing to do, but I can get behind that kind of thinking. Thomas was an apostle, but he didn't believe that Jesus truly was God until he actually saw Jesus resurrected with nail marks... did Jesus love him? Ya, totally. Doubt isn't a sin... explore faith and test it.

When I was a kid I'd get into trouble because I listened to heavy metal. I was supposed to listen to pop or light rock. What I thought was interesting was that my parents wanted me to listen to "normal music" which was all about sex, love, self interest, especially in the 80's. Even though death metal focused on Satan, that was more Christian to me than Paula Abdul... Satan is real, their talking about Satan and how we wants to destroy us all and inflict suffering... Slayer seemed more biblical than Whitesnake. :)

When Open Source came along it jived for me because it parallels Christianity. Community and working together and evangelizing... these are integral Christian concepts that I immediately got. God has called me to be a SysAdmin and He instructs us to love our brother as ourselves, helping one-another in love and charity... why do you think I write so much? :)

God has faithfully guided me through my entire life, and I trust Him implicitly. He's proven it time and time again. I get a raise, a month later a new bill comes in. I get a bonus, 2 weeks later the car breaks down. I get downsized and before the day is out I have an offer to start immediately. I commonly say that "God opens doors for people... but I'm not very bright, so He just kicks me through 'em." When adversity or change happens I don't get scared that I'm screwed... I look for what God is doing. With my new house... a friend tells me about the cheap foreclosures in Tracy and we start researching it, one week later we get our 60-day notice to leave our rental in Fremont, 2 weeks later we find the right house, 3 weeks later we close escrow, 1 week later we're moved in, and we have just enough time to clean out the old place before we have to give it back. Now, if you take these sorts of things and call them random chance or coincidence than your faith in chaos is stronger than my faith in Jesus. :)

I'm not perfect.... definitely not. I've probly got more problem than any of you reading this. I'm a sinner and I know it... but I want to change, and Jesus explains how, through faith and repentance. Heaven is great and all, but frankly I'm concerned about how to live now... Christianity is about both now and forever at the same time.

I won't post any more about Christianity directly in this blog because I'm aggregated fairly widely and don't want to get yanked for being off topic, but its important to me and now seemed like the time to share a bit.

If you are interested in Christianity, either as a fellow believe or as someone who's just checking things out, I highly recommend you check out Mars Hill Church (Mark Driscoll's church in Seattle, not Rob Bell's church in Michigan). They just finished a 13 part "Doctrine: What Christians Should Believe" series which is totally awesome!! No fluff, just great information for a great young preacher.

To see a short introduction before jumping in, Watch the Preview to the Doctrine Series.

I highly encourage you to look through the various sermon series, all in video. Plus, you can subscribe to the Video Podcast on iTunes. Mars Hill is the first church I've seen that actually feels like an apostolic church, young new believers with real questions. Want to find a church that actually talks about sex? Answers questions like "Is oral sex ok?"... seriously, amazing church for real people with real questions and real interest. Check it out!

If you're already a Christian, but you've fallen away from your walk, there are lots of great ways to get re-engaged and centered. One thing that helps me is attacking the Bible in new ways. Try a new translation! I still love the King James Tranlation (KJV AV), but also have found that NKJV, ESV and NASB (NASB being the most literal translation available) are really wonderful and accurate. Remember, the KJV is not the word of God, its an english translation of it... don't let people shoe-horn you into a single translation. I do, however, advise staying away from paraphrases such as "The Message" except for cross-reference. Seriously, ESV is easy to read and accurate.

Another great resource is audio bibles. Its hard to sit down and read an entire book without getting hung up on phrases or specific stories. Get a broad view of the scriptures!! I've found that when you just listen to the New Testament from beginning to end you walk away with a very different vibe than when you just work on given passages. I use this one:

I prefer to refer to this as "The Bible as read by Darth Vader". :)

Lastly... for new Christians and old alike, I offer one important piece of advice that I myself have struggled with in the past. Focus on Jesus! Christianity is about... duh... Christ. When you focus on "god", which god? Its impersonal and abstract, and the Bible is anything but impersonal or abstract. Jesus is God, Jesus is alive, Jesus loves you and died for your sin. Whether you believe that or not, focus on Jesus, not "god". Pray, calling out to Jesus by name, and talk to Him... see what happens.

It sucks that we're forced to leave Fremont, and it sucks that we'll be so far away... but we're insanely blessed none the less, given that we now own our first home which is significantly nicer than our current rental, we're still living in California and close to the Silicon Valley rather than having to leave the state like so many other native Californian's have, and a new locale means new experiences, new friends, a new church, and opportunities. We've leaned on the Lord 110% and He's had our backs the whole way, everything just happened easily and quickly and we're thankful for His provision, yet again.

A change of scenery is always a useful thing. I'm busy atm learning about tile, caulking, and plumbing, but hope to get down to more interesting things soon so that I can get some fun content into this blog.

I've been privileged to know several of the great dads in the Sun/OpenSolaris ranks. Dr. Stephen Hahn, Jeff Bonwick, Bill Moore, Paul Armstrong (Google), and Chris Baker are all absolutely first rate fathers, aside from being brilliant technologists. I am immensely thankful for the opportunity to know not just these great men but also their families.

To all the fathers out there, a happy fathers day.

Of course, there is always debates over leaks... is it real or faked, who knows, but it makes me hope none-the-less. If I could get one with 32GB at $499 I'll be a very happy camper. In the mean time I'm restricted to my Joyent BlackBerry.

UPDATE: And its here! No word about iChat AV, but the price has been greatly reduced, GPS and 3G are there. I'll be getting one as soon as it releases! :)