Phoronix has quickly risen as my favorite review site (along with Anandtech). They focus more on the *NIX world than other sites and have set the bar for a benchmark review site. In particular thanks to the Phoronix Test Suite, a venerable suite of benchmarks beautifully weaved together. Did I mention they like OpenSolaris? That helps too. 🙂

They recently uploaded a presentation by site founder Michael Larabel entitled: The Five Stages of Benchmark Loss. In it Michael shares his collected experiences on how people react when they dislike benchmark results. As he says, when someone comes out the winner in a benchmark they naturally exclaim, “Of course, we’re awesome, thanks.” but if they loose all kinds of pain results, which he lays out into 5 stages.

The five stages are:
1. Shock
2. Denial
3. Discreditation
4. Analysis
5. Acceptance

This is, imho, a must listen to recording. It’s audio only, the quality is horrible, but get some headphones and battle through. It’s absolutely worth it.

This is particularly interesting to me because it strikes at the heart of one of my greatest irritations with the geek world; that is, a total dismissal of benchmarking. Its paradoxical really, as computer scientists we should be preforming experiments with measurable results, analyzing data, using and constructing new tools, etc. But, any time a benchmark result is posted so many dipshits simply exclaim “bullshit, your an idiot” that it makes people incredibly gun shy.

I believe the result is the world of horrible benchmarking tools we have (particularly in the UNIX world) and almost no information on how to effectively benchmark systems and storage. If only we would help each other out by leaving Michael’s Stages 2 & 3, “Denial” and “Discreditation” and instead move directly to Stage 4, “Analysis”, the world would be a much better, kinder, and more informed place.

For instance, “bonnie++” is said to suck. But how many people can tell you why? “iozone” is said to be great. Again, how many people can tell you why? At some point, most particularly in this area, FUD is ingested and regurgitated on folks just trying to learn something, who in turn become bitter and pass on the pain to a whole new generation.

It is, I believe, the single best example of cynical bitterness and stupidity that rages our industry. To be sure it’s part of a larger issue, wherein geeks famously trash this technology or that (consider opinions against Java as another example) which are based in outdated or incomplete information or understanding. It only serves to discourage folks and keep the viral spread of cynicism and stupidity going.