Archive for April, 2010

Splunk 4.1 is a Winner

Monday, April 19th, 2010

Splunk 4.1 has arrived and really raises the bar for an already amazing tool. Several new key features have been added that make it worth an immediate upgrade:

  • Integration with Industry Single Sign-On products, such as OpenSSO, Tivoli, Oracle Identity Management, etc.
  • Event Level Workflows, such as opening a ticket or taking some action
  • PDF Reports
  • Event Extraction
  • Live Dashboards and Views
  • Real-time Search

The big killer-app for me that takes Splunk to a whole new place is the Real-time Search feature. You can enter a search string like usual and then set a time period, such as 1 minute. Then you will see events rolling in, real-time. At the end of the specified period the results clear and you see the next period of results start rolling in real time.

This feature is just absolutely amazing… its like an intelligent web-based analytical tail/grep of your logs! This is great for not just identifying problems but also verifying that you’ve fixed a problem or for use during debugging sessions. Did that config change really stop the 404′s? Have those errors actually stopped coming or do you need to keep working? This feature is a must-have for all sysadmins.

You can check out the list of new features and see videos about them on the What’s New in Splunk 4.1 page.

Posted in cuddletech | 71 Comments »

The DTrace fsinfo Provider

Monday, April 19th, 2010

If you’ve seen one of my talks on ZFS Tuning you’ve witnessed my love for the fsinfo provider (and the fsstat command as well) first hand. This amazing but poorly documented provider can give you some great high level insight into your I/O pattern. If you’ve ever used truss just to watch iops, then give this a whirl instead.

There are probes for each VFS operation: 

   ID   PROVIDER            MODULE                          FUNCTION NAME
14937     fsinfo           genunix                       fop_vnevent vnevent
14938     fsinfo           genunix                       fop_shrlock shrlock
14939     fsinfo           genunix                    fop_getsecattr getsecattr
14940     fsinfo           genunix                    fop_setsecattr setsecattr
14941     fsinfo           genunix                       fop_dispose dispose
14942     fsinfo           genunix                       fop_dumpctl dumpctl
14943     fsinfo           genunix                        fop_pageio pageio
14944     fsinfo           genunix                      fop_pathconf pathconf
14945     fsinfo           genunix                          fop_dump dump
14946     fsinfo           genunix                          fop_poll poll
14947     fsinfo           genunix                        fop_delmap delmap
14948     fsinfo           genunix                        fop_addmap addmap
14949     fsinfo           genunix                           fop_map map
14950     fsinfo           genunix                       fop_putpage putpage
14951     fsinfo           genunix                       fop_getpage getpage
14952     fsinfo           genunix                        fop_realvp realvp
14953     fsinfo           genunix                         fop_space space
14954     fsinfo           genunix                        fop_frlock frlock
14955     fsinfo           genunix                           fop_cmp cmp
14956     fsinfo           genunix                          fop_seek seek
14957     fsinfo           genunix                      fop_rwunlock rwunlock
14958     fsinfo           genunix                        fop_rwlock rwlock
14959     fsinfo           genunix                           fop_fid fid
14960     fsinfo           genunix                      fop_inactive inactive
14961     fsinfo           genunix                         fop_fsync fsync
14962     fsinfo           genunix                      fop_readlink readlink
14963     fsinfo           genunix                       fop_symlink symlink
14964     fsinfo           genunix                       fop_readdir readdir
14965     fsinfo           genunix                         fop_rmdir rmdir
14966     fsinfo           genunix                         fop_mkdir mkdir
14967     fsinfo           genunix                        fop_rename rename
14968     fsinfo           genunix                          fop_link link
14969     fsinfo           genunix                        fop_remove remove
14970     fsinfo           genunix                        fop_create create
14971     fsinfo           genunix                        fop_lookup lookup
14972     fsinfo           genunix                        fop_access access
14973     fsinfo           genunix                       fop_setattr setattr
14974     fsinfo           genunix                       fop_getattr getattr
14975     fsinfo           genunix                         fop_setfl setfl
14976     fsinfo           genunix                         fop_ioctl ioctl
14977     fsinfo           genunix                         fop_write write
14978     fsinfo           genunix                          fop_read read
14979     fsinfo           genunix                         fop_close close
14980     fsinfo           genunix                          fop_open open
    The arguments to the probes are:

  • args[0]: fileinfo_t *
  • args[1]: Return value (0 for success, in the case of writes this is the write size in bytes (ssize_t)).

So lets look at the most generic script we could write: 

#pragma D option quiet

fsinfo:genunix::
{
printf("%s (%s) %s: %d [%s]n", probename, execname, args[0]->fi_pathname, args[1], args[0]->fi_fs);
}

The output specifies the probename (operation), process name that generated the op, the pathname if applicable, the return value, and finally the filesystem type: 

$ dtrace -s fsinfo.d
lookup (idmapd) /etc: 0 [ufs]
lookup (idmapd) /etc/resolv.conf: 0 [ufs]
getattr (idmapd) /etc/resolv.conf: 0 [ufs]
poll (idmapd) <unknown>: 0 [sockfs]
poll (idmapd) <unknown>: 0 [sockfs]
poll (idmapd) <unknown>: 0 [sockfs]
...
close (mysqld) /local/zone/root/var/tmp/#sql_47d3_0.MYI: 0 [zfs]
close (mysqld) /local/zone/root/var/tmp/#sql_47d3_0.MYD: 0 [zfs]
lookup (mysqld) /local/zone/root/var: 0 [zfs]
lookup (mysqld) /local/zone/root/var/tmp: 0 [zfs]
...

This can quickly be jazzed up by adding a conditional by zonename or filesystem type or process. We could do some aggregations rather than the play-by-play.

While the fsstat command can give you per-operation counts by mount, this gives you a lot more data which in turn can be used to find unusual I/O’s that you didn’t expect or help you construct realistic benchmarks using tools like FileBench.

Posted in OpenSolaris | 73 Comments »

iPad Review

Sunday, April 4th, 2010

DISCLAIMER: I say “iPhone” when I technically should say “iPod Touch”. iPad has no mic and no phone. But, I say iPhone because thats the device most of us have and use for real applications. You know what I mean.

One of the perks of working for a jazzy startup like Joyent who’s had a good year: free iPad’s for all. Our Christmas bonus from Joyent. w00t. So I pre-ordered mine and got it today, as promised by Apple.

So, lets get down to brass tax: its an oversized iPhone. No more, no less. There is no expansion slot for storage, just the same buttons and I/O port as the iPhone. It looks the same, feels the same, works the same.

Now, bearing that in mind. There are some things that iPhone could do, but it was a bit ridiculous as iPhone size. My favorite example is the iSSH application which allows you to SSH to a server. Its a great app, but on an iPhone its so small its only a gimmicky sort of app that might be useful in a dire emergency but otherwise is just for show. VNC apps are the same. Even some financial apps with charts… they work on iPhone but its just too small. These are the apps that shine on iPad, provided there is an upgraded version which natively takes advantage of the screen real estate.

The same goes for video. Video on iPhone is fine, but its really tiny. Given the choice between viewing on iPhone or your laptop, you’ll choose your laptop every time. With iPad the size is perfect for viewing and the portability is great.

Again, this applies to books/pdf’s, etc. The iPhone was just too small. With iPad you have plenty of viewing area and its more comfortable because of the larger fonts and more text per page.

If you, like myself, have wanted a device to carry all your PDFs on, this is definately it. Download the “GoodReader” app. You can transfer PDFs to your iPad using a variety of methods and view them from a library. I’ve loaded all the Solaris, NIST, CIS, etc manuals and several PDF books I’ve downloaded from Safari to it and they look great.

There have been two big concerns about iPad, so lets dig into those.

The first is that its backlit. Meaning, its harsh on the eyes and therefore not a viable eBook reader. This is true as expected. The brightness is on par with the iPhone and while reading news and skimming books is fine, reading for hours on end will burn out your eyes just like a monitor.

The second is about the usability of the on-screen full size QWERTY keyboard. I was shocked! It works beautifully! When I typed my first sentence I expected to have gotten every other letter wrong, but to my surprise it was spot on the first time. The only trick is to not think about it… just touch type and don’t look, and you’ll be fine. I’m sure there is some magic in there doing minor corrections based on what you meant to press versus really did, but whatever they did it is brilliant. Absolutely positively no issues typing on it at full speed.

So, should you fork out $500 for one? IMHO, no. At least, not yet. When all the apps are iPad native, it’ll be great, but for now it needs more time. This is only day one, and they are converting quick, but give it time.

If you are an iPhone addict and use it for news, video and browsing a lot, then you absolutely will love the iPad. However, if you primarily use your iPhone as a phone, use the maps, and maybe use some apps for checking your bank account, then wait. That is to say… if you use your iPhone as though it weren’t a mobile device, then congrats, iPad is what you’ve wanted.

The biggest downer is that many of the apps are releasing “Tablet Versions”, where you have to buy the new app, typically at double the price. “Things” for iPad is a new purchase and is $20. Thats just a joke. iSSH, thankfully, was a free upgrade. Watch that space carefully. Also, more serious business apps are coming, such as OmniGraffle, but its a $50 app! That’s insane, imho. You’ll see the app wars really come alive in the next couple months.

Posted in cuddletech | 73 Comments »