I’ve been watching and silently participating in the industry trend known as “DevOps” for some time now. I’ve dragged my feet on writing about it for reasons that I won’t go into now, but this blog post, which made it to slashdot, has pulled me out of my shell. Ted Dziuba picks a fight by saying:

“This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them – Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don’t overthink things, and arrive at simple, scalable solutions.”

So Ted wants to sensationalize his amazing realization that basic UNIX tools are actually useful by dumping on Devops. Fantastic.

Slashdot includes a link to Patrick Debois What Is This Devops Thing, Anyway? which Ted does not. While Patrick is rightly credited as kicking this movement off and dubbing the term DevOps, his blog entry describing it is vague, at best. Ted summed it up, wrongly, as “SA’s start writing unit tests and other things to help developers warm up to them”. Wrong wrong wrong.

So lets break this all down and get to some reality.

DevOps is a buzzword which identifies a new cultural evolution in systems administration. It comes from the WebOps world where the divide between programmers who write the software (“dev”) and sysadmins who deploy and manage the software (“ops”) is paper thin. Patrick put a buzzword and some conferences around the growing trend of sharing methodology across this boundary.

So, the first manifestation of “devops” is a world in which developers learn from operations and operations learns from developers. The result is ops teams using Hudson for continuous integration testing, unit testing everything produced in ops just like dev does, always pushing trunk, allowing features to be enabled or disabled in software without having to rollback code, etc. So, in this world, the two groups act much more uniformly in process and tools.

But, the second manifestation is more interesting. Administrators outside the webops space are being caught up in the momentum of the movement and listening to the same talks but hearing different things. This is currently mostly affecting SMB and startups which have been, for the last 3-4 years, greatly influenced by the webops shops.

To this latter group, Devops is really “Next Generation ITIL; ITIL for the masses”. They hear the Devops guys say things like “Operations needs to serve the business, not dwell in a cave of geeky solitude” or “Operations needs to measure everything and have release process, change management, source control, etc.” What a lot of people are hearing is “ITIL is kool again”. Not just ITILv3, but also CobiT, CMMI, ISO20K, ISO27K, etc. Because the concepts that really strike a cord with many systems administration teams are clearly defined in ITILv3. For those who find ITIL too big to chew off, the book Visable Ops by Tripwire founder Gene Kim, et al, has become a cult classic in how to implement ITIL in a realistic and sensible approach.

The reason this push for ITIL is different than the first, is that now ITIL is being adapted and pushed from the bottom up, as opposed to the first wave which was forced from the top down. Engineers, not executives, are thinking “Ya, change management sounds good, but how do I do that?” and they find ITILv3 already spells it out. The effect is that ITIL is now what it was intended to be, a collection of best practice which can be used as a reference by organizations to augment and improve existing practices. The first wave was bastardized by CIO’s chucking out the baby with the bathwater so they could look busy. But I digress…

The other side is that we’re seeing a real push in devops for a unified toolchain. Check out the Devops Toolchain Project. The open source administration tools are getting really solid. This toolchain breaks down into the areas of monitoring, provisioning, configuration management, and command and control. The movement to standardize as an industry on CFengine, Puppet or Chef, has cemented the second brick of the foundation. Improving command and control solutions (ControlTier, Nanite, mcollective, etc) are getting very close. The important result is an industry push to standardize on a similar set of tools thus providing re-use between organizations.

Finally, sprinkle Agile in to taste. A little SCRUM or Kanban, a touch of Jira, a little Git Hub, a pinch of manifesto, shake vigorously to combine.

Whats the catalyst though? Why now? None of this is really new, so why do we suddenly seem to be getting our act together as an industry?

Cloud. We’ve all been told for years that the number of machines we’re managing is going to explode and, as is did, our methods would have to change. What we saw was a slow ramp and with it we slowly adapted traditional methods to increase numbers of resources. However the traction that cloud computing has brought finally set off the explosion of systems.

Let me use an age old analogy you’ll all be familiar with. If you have to do something less than 10 times, big deal, do it manually. If you have to something more than like 10 times you might as well spend the time writing a script to automate it. But if you have to do it 100 times, then you really have to get your act together and build a reasonably robust solution.

Cloud pushed us as an industry from 10 to 100. Writing piles of ad hoc scripts won’t cut it anymore because you don’t know what your environment will look like in a year. It might be bare metal boxes, it may be virtualized in house using VMware, or it may be a cloud solution based on Joyent, AWS, and Rackspace. Whatever it is, you have to change the way you approach old problems of provisioning, configuring, monitoring, and managing systems.

So “systems administration” starts maturing into “operations”. Process is more important. Policy is more important. Standardization on generic tools like Puppet or Chef is more important. IT can no longer be an illusive crack squad of hackers that do things in a way that ensures both l337-ness and job security. IT has to be scalable as an organization and is more and more becoming a real revenue generating part of the company that has to actually act like part of the company by answering to the board like any other business unit.

All these things are colliding together. And at just the right time in history, Patrick Debois comes along and says “Devops” and proposes rethinking the way we do operations. No one really cares what Patrick says after that because so many of us have been rethinking all these things and just didn’t have a word to put to it. “Devops” is the word.

Words have power. Thanks to “Devops” people are now coming together, sharing together, learning from each other. We’re not alone it seems, which means we all don’t have to re-invent the wheel.

Web 2.0 might have been a crappy buzzword, but it marked a cultural change in what users expect from their web experience. Web 2.0 was a cultural evolution. Cloud might have been a crappy buzzword, but it marked a cultural evolution in how we architect our infrastructure, from a physical model to a virtual model. In the same way, “Devops” is the cultural evolution of what is expected from IT and Systems Administration. Its been a long time coming and all the pieces are now aligning and as an industry we’re finally maturing. Focus on the “culture of change” or you’ll just miss the whole point.