Devops Demystified

25 Oct '10 - 20:06 by benr

I've been watching and silently participating in the industry trend known as "DevOps" for some time now. I've dragged my feet on writing about it for reasons that I won't go into now, but this blog post, which made it to slashdot, has pulled me out of my shell. Ted Dziuba picks a fight by saying:

"This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions."

So Ted wants to sensationalize his amazing realization that basic UNIX tools are actually useful by dumping on Devops. Fantastic.

Slashdot includes a link to Patrick Debois What Is This Devops Thing, Anyway? which Ted does not. While Patrick is rightly credited as kicking this movement off and dubbing the term DevOps, his blog entry describing it is vague, at best. Ted summed it up, wrongly, as "SA's start writing unit tests and other things to help developers warm up to them". Wrong wrong wrong.

So lets break this all down and get to some reality.

DevOps is a buzzword which identifies a new cultural evolution in systems administration. It comes from the WebOps world where the divide between programmers who write the software ("dev") and sysadmins who deploy and manage the software ("ops") is paper thin. Patrick put a buzzword and some conferences around the growing trend of sharing methodology across this boundary.

So, the first manifestation of "devops" is a world in which developers learn from operations and operations learns from developers. The result is ops teams using Hudson for continuous integration testing, unit testing everything produced in ops just like dev does, always pushing trunk, allowing features to be enabled or disabled in software without having to rollback code, etc. So, in this world, the two groups act much more uniformly in process and tools.

But, the second manifestation is more interesting. Administrators outside the webops space are being caught up in the momentum of the movement and listening to the same talks but hearing different things. This is currently mostly affecting SMB and startups which have been, for the last 3-4 years, greatly influenced by the webops shops.

To this latter group, Devops is really "Next Generation ITIL; ITIL for the masses". They hear the Devops guys say things like "Operations needs to serve the business, not dwell in a cave of geeky solitude" or "Operations needs to measure everything and have release process, change management, source control, etc." What a lot of people are hearing is "ITIL is kool again". Not just ITILv3, but also CobiT, CMMI, ISO20K, ISO27K, etc. Because the concepts that really strike a cord with many systems administration teams are clearly defined in ITILv3. For those who find ITIL too big to chew off, the book Visable Ops by Tripwire founder Gene Kim, et al, has become a cult classic in how to implement ITIL in a realistic and sensible approach.

The reason this push for ITIL is different than the first, is that now ITIL is being adapted and pushed from the bottom up, as opposed to the first wave which was forced from the top down. Engineers, not executives, are thinking "Ya, change management sounds good, but how do I do that?" and they find ITILv3 already spells it out. The effect is that ITIL is now what it was intended to be, a collection of best practice which can be used as a reference by organizations to augment and improve existing practices. The first wave was bastardized by CIO's chucking out the baby with the bathwater so they could look busy. But I digress...

The other side is that we're seeing a real push in devops for a unified toolchain. Check out the Devops Toolchain Project. The open source administration tools are getting really solid. This toolchain breaks down into the areas of monitoring, provisioning, configuration management, and command and control. The movement to standardize as an industry on CFengine, Puppet or Chef, has cemented the second brick of the foundation. Improving command and control solutions (ControlTier, Nanite, mcollective, etc) are getting very close. The important result is an industry push to standardize on a similar set of tools thus providing re-use between organizations.

Finally, sprinkle Agile in to taste. A little SCRUM or Kanban, a touch of Jira, a little Git Hub, a pinch of manifesto, shake vigorously to combine.

Whats the catalyst though? Why now? None of this is really new, so why do we suddenly seem to be getting our act together as an industry?

Cloud. We've all been told for years that the number of machines we're managing is going to explode and, as is did, our methods would have to change. What we saw was a slow ramp and with it we slowly adapted traditional methods to increase numbers of resources. However the traction that cloud computing has brought finally set off the explosion of systems.

Let me use an age old analogy you'll all be familiar with. If you have to do something less than 10 times, big deal, do it manually. If you have to something more than like 10 times you might as well spend the time writing a script to automate it. But if you have to do it 100 times, then you really have to get your act together and build a reasonably robust solution.

Cloud pushed us as an industry from 10 to 100. Writing piles of ad hoc scripts won't cut it anymore because you don't know what your environment will look like in a year. It might be bare metal boxes, it may be virtualized in house using VMware, or it may be a cloud solution based on Joyent, AWS, and Rackspace. Whatever it is, you have to change the way you approach old problems of provisioning, configuring, monitoring, and managing systems.

So "systems administration" starts maturing into "operations". Process is more important. Policy is more important. Standardization on generic tools like Puppet or Chef is more important. IT can no longer be an illusive crack squad of hackers that do things in a way that ensures both l337-ness and job security. IT has to be scalable as an organization and is more and more becoming a real revenue generating part of the company that has to actually act like part of the company by answering to the board like any other business unit.

All these things are colliding together. And at just the right time in history, Patrick Debois comes along and says "Devops" and proposes rethinking the way we do operations. No one really cares what Patrick says after that because so many of us have been rethinking all these things and just didn't have a word to put to it. "Devops" is the word.

Words have power. Thanks to "Devops" people are now coming together, sharing together, learning from each other. We're not alone it seems, which means we all don't have to re-invent the wheel.

Web 2.0 might have been a crappy buzzword, but it marked a cultural change in what users expect from their web experience. Web 2.0 was a cultural evolution. Cloud might have been a crappy buzzword, but it marked a cultural evolution in how we architect our infrastructure, from a physical model to a virtual model. In the same way, "Devops" is the cultural evolution of what is expected from IT and Systems Administration. Its been a long time coming and all the pieces are now aligning and as an industry we're finally maturing. Focus on the "culture of change" or you'll just miss the whole point.

- - C O M M E N T S - -

As a sysadmin at a shop where we have moved from 50 to 350 servers in the past few years and split from one engineering department, to 2, and now into 5 that we’re still drawing the borders on to get a better hold on the alpha to production cycle I’m grateful to see lots of other people getting excited and sharing about devops stuff in a useful way. We already use build and config automation systems heavily but have cooked up a lot of our own l33t duct-tape all around them to keep everything working. We’re at a point that metrics and validation at every step of the way is becoming crucial and I’m super stoked to see some of the resources people are sharing and really excited to start integrating some of them into our environment.

Ben, thanks again for the clarity and some great links.

Mike (Email) - 26 October '10 - 03:46

” No one really cares what Patrick says after that because so many of us have been rethinking all these things and just didn’t have a word to put to it. “Devops” is the word.” Hahahaha :) very well put, Ben :) This is just the kind of realization I went through some weeks ago when I asked on our company’s internal mailing list about just what “devops” meant to us :)

Sriram Narayanan (Email) (URL) - 26 October '10 - 18:16

Always great to hear your particular take, Ben. Thanks for the thoughtful post. It’s an exciting time seeing system administration grow up.

Casey - 26 October '10 - 19:11

ITIL and CMMI never went out of fashion to begin with. It’s just a matter of how serious about quality and clearly defined processes and repeatability the workplace is.

Better workplaces have those nailed down pat. Crystal clear.

UX-admin (Email) - 27 October '10 - 14:49

Hi Ben,

I have been one on the outside catching up “on the devops thing”.
Reason being simply that I’m paid to run a realworld business, not a web shop.

I welcome the devops idea a lot, as I see many admins and devs end up in closed holes builing all the same for their specific company but no real technical exchange ever takes place. (i.e. no 20 company admins will sit and discuss their leetnuks ks profiles. But they should!)

I just felt like throwing up when I read a few more presentations with statements like “Ops people just get out of the way”

I can complete that sentence: “because we got the cool new things that will cure everyone. They’re not yet tested, and we’re only 3 years into this, but we’re cool devs and reaching 5 nines uptime is impossible anyway. My chum tried and it didnt work”

I think the separation between devs and admins didnt always exist and it was added for a reason..

I am afraid the whole devops thing will go down once it faces companies within the “old economy”, not because culture admins put something in it’s way, but because the people on the devops train are so over-proud of their idea (as if it were new. met the infrastructure admins for once?) that they race past the people that are warning them of “end of track” signs ahead.

really, how do people end up thinking they need not value prior experience?
there’s a difference between successfully running 5000 amazon ec2 instances and running one enterprise server.
Both sides should care to learn – thats why I read up their presentations and toy with mcollective and all kool stuff…

darkfader (Email) (URL) - 31 October '10 - 12:17

@darkfader; If you’ve seen the presentations you’ll have noticed the same thing I have… This really has very little do with “dev”. Sure, people say they want to bring “dev” and “ops” closer together, but they don’t mean a single team. 98% of all presentations are about a deeper ops environment that isn’t afraid of new technologies and embracing standards.

At Joyent we’re not necessarily doing the “devops” thing, as some people might think of it. At Joyent we are all engineers, working on engineering problems, each with a special problem domain. There is no big love in, just engineers working independently and corporately on solving problems in the most sensible ways possible in conjunction with the best practices of our respective fields.

I think we’d both agree that there are some people that are getting the wrong idea from the term, just as people did from cloud. But I think we’d also agree that good things are happening and our industry is maturing as a result of both. And thats a good thing, no matter how it comes. ;)

benr - 01 November '10 - 04:54

@darkfader – And what about those of us from that enterprise world who think DevOps is a good idea? I’ve never managed 5000 EC2 instances but I’ve managed mission-critical infrastructure that transacted billions of dollars a day and required 5 nines uptime.

DevOps is about far more than WebOps. To me it’s a convenient label that I am going to steal to explain to “old economy” companies that “because we’ve always done it like this” isn’t always the right approach. If it takes DevOps to do that – excellent. If I can steal some Visible Ops thunder I’ll do that. If I have to invent YodaOps to get the message across then I’ll do that too.

James Turnbull (Email) (URL) - 03 November '10 - 21:25

Great ….You have beautifully presented your thought in this blog post.

rugs for sale (Email) (URL) - 19 November '10 - 05:59


electronic components (Email) (URL) - 22 November '10 - 01:36

Xiling auction held two consecutive manuscripts ugg classic short boots of modern celebrity auction, two were 100% of the turnover rate of turn than the opening price turned ugg classic tall boots the turnover number is considerable. Hu first of “trying to set the second set of” script to create modern single-volume manuscript at the highest price, while the ugg classic mini boots spring auction in charge and the preceding paragraph Chang ugg classic cardy sale Shen, Bing Xin, works to maintain full personal best in the market results. The autumn auction, we again focused on the introduction and the same number ugg bailey button sale of old family collection of the preceding paragraph, not only in the scope of the ugg bailey button triplet boot market has been well known in the contents of the Letters, the identity of the preceding paragraph have more stringent requirements, but not limited to the scope of calligraphy ; the same time, also try to introduce some in the political, scientific, ugg roxy short boots economic, cultural and other celebrities of the manuscript ink, even in today’s market, their prices are not yet a complete expression, but it ugg roxy tall sale really marked a “celebrity handwriting,” the significance of the . Believed to nourish the author in the original disposition of Hangzhou, will return to their proper value. [[]]

ugg classic short (Email) (URL) - 24 November '10 - 01:23

It’s funny how we adopt words and adapt our lexicon to the times. This is a very useful slant on things.

Wholesale NFL jerseys (Email) (URL) - 02 December '10 - 09:03

I gave my site a few examples below. If you appreciate my comments in you enter.

Cheapest Football Jerseys (Email) (URL) - 02 December '10 - 09:03

The blog article very surprised to me! Your writing is good. In this I learned a lot! Thank you!

Red Bull Hats (Email) (URL) - 02 December '10 - 09:04

I mean I just read through the entire article of yours and it was quite good but since I’m more of a visual learner,I found that to be more helpful well let me know how it turns out. This is good…thanks for sharing

bike parts (Email) (URL) - 04 December '10 - 03:20

The blog article very surprised to me! Your writing is good. In this I learned a lot! Thank you!

Cheapest Football Jerseys (Email) (URL) - 06 December '10 - 07:36

This article is wonderful. Now i fall in love.

moncler (Email) (URL) - 07 December '10 - 07:32

Thanks for post and other tips. Now I’m making my own design.

ugg boots uk (Email) (URL) - 07 December '10 - 07:32

Ugg boots online , Ugg boots online
UGG boots women , uggs women boots
Ugg boots sale , Ugg boots sale
UGG bailey botton boots , UGG bailey botton boots
UGG women boots UK , UGG women boots UK

UGG boots (Email) (URL) - 07 December '10 - 09:01

Hua 101211 Once we reach there, so many wonderful north face Denali hoodie dreams will come true and the pieces of our lives will be fit together like a completed jigsaw puzzle. How restlessly we pace the north face jackets sale aisles, damning the minutes loitering, waiting, waiting, and waiting for the station. [[]]

north face denali sale (Email) (URL) - 11 December '10 - 09:33

thanks for sharing! welcome to my website

new era hat (Email) (URL) - 13 December '10 - 05:37

Any way Ill be subscribing to your feed and I hope you post again soon.thanks for sharing good

rug (Email) (URL) - 15 December '10 - 11:16

Hua 101220
The authors of the Designer leather handbag American Declaration proclaimed happiness to be one of the essential rights that make life worth living. William James, the first Mulberry bags uk American psychologist, echoed this emphasis on happiness: If we were to ask the question:What is human life’s chief concern? One of the Designer bags sale answers we should receive would be: It is happiness. Every philosophical and religious system has offered its pathway to happiness for the individual and the Mulberry alexa bags group. Happiness has been related to pleasure, refuge from pain, intellectual contemplation, union with God, friendship, children,Gucci bags 2010 wealth, honor, successful activity, and even state burial with statues! Today, scientists have joined the dialogue to seek answers about what contributes to happiness or Mulberry bags sale, as they call it, subjective well-being. [[]]

chanel bags (Email) (URL) - 20 December '10 - 05:24


ghd outlet (Email) (URL) - 27 December '10 - 03:04

It’s funny how we adopt words and adapt our lexicon to the times.

wholesale (Email) (URL) - 30 December '10 - 07:29

I enjoyed reading this post and i have to say i found it veru informative,i wish there was a newsletter i could subscribe to as i would love to get updates from this blog.

massjo (Email) (URL) - 21 January '11 - 06:41

You would not believe how long I’ve been looking for something similar to this. Looked through 5 pages of Google results and couldn’t find anything good.

rugs (Email) (URL) - 21 January '11 - 11:53

sgarealestates provide you a comprehensive range of good quality real estate projects by all the reputed builders be it residential, commercial and retail . Besides we also serve you in leasing,selling, reselling of your properties and enjoying the returns there on. We offers you best deal of property in chandigarh, property in panchkula, property in zirakpur and property in mohali on reasonable rates.

property in chandigarh (URL) - 09 February '11 - 10:09

Personal information

Remember your information?

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.