Lets take a little journey through a thought and see where it shall take us.
Let us first consider “devops”. It is a cultural movement which codifies various characteristics, including the zen like interweaving of development and operations, better integration of “IT” with the business objectives as a whole, infrastructure as code, agile (or lean depending on who you speak with) operations, etc, etc. Alright. As I’ve stated in the past, its all the things ITIL claimed to be but somehow more hip. Alright, capture that in your minds eye and then move it aside.
Next consider the title “Systems Engineer”. It is a very ambiguous title to be sure, because it is cross discipline. Borrowing from WIkipedia: “Systems engineering is an interdisciplinary field of engineering that focuses on how complex engineering projects should be designed and managed.” OK… now, lets break it down into its consituent parts. An engineer is, Wikipedia says: An engineer is a professional practitioner of engineering, concerned with applying scientific knowledge, mathematics and ingenuity to develop solutions for technical problems.” Alright, and “system” comes from Greek literally meaning: “whole compounded of several parts or members, system”. Great, now put it back together… a System Engineer is one who solves technical problems using science and math to to make diverse components act harmoniously together as one. Great, put that aside.
As a geek, I have two pictures of the classical “systems engineer”. The first is a guy at NASA building the lunar lander, pocket protector and slide rule in hand, he’s making sure he can cram everything he needs into a landing module while still fitting on top of a rocket and capable of re-entry into the atmosphere. This guy has to think outside the box, consider every detail and yet be flexible enough to change the plan, like splitting the lander into two pieces, one which stays on the lunar surface and the other that blasts back to earth, etc. The other guy is a “kernel programmer”, I put that in quotes because maybe they might build compilers or drivers or libraries, but these folks know their ASM and don’t think FORTH is out of style yet. Most of these guys are up there in years, work at stodgy companies, and don’t have much of a sense of humor. Externally these guys don’t seem much different… ones at AT&T or IBM, the other NASA, but the same image:
Lets now consider OS’s and computers. On the whole, OS’s aren’t changing. Is Linux fundamentally that different than other UNIX’s? We’re not seeing radical shifts in what we use. Similarly, in the chip space we’re all now comfortable with the reality that 5GHz CPU’s aren’t going to pop up any time soon (at least, not in general use) but we’re going to get more and more 2.xGhz cores. 2, 4, now 6, etc. Even your parents little Dell now has 2 or more CPU’s, soon your kids will have a laptop with 6 or more… and they won’t know or care. The point is that we’ve hit a maturity point in which the building blocks are pretty solid, but the way in which we use them is now more interesting. The old image of the Systems Engineer feels, well, old.
OK… lets now knit this together.
When I first came in contact with the “Devops” movement I thought it was about better systems administration, a maturing of the craft. It was, I thought, more about Ops and less about dev. Dev had their day in the sun with Agile… now sysadmins were getting theirs. Remember, if you can, when Agile came on the scene and how giddy developers got with eXtreme Programming, then Pragmatic Programmer books are everywhere, and then SCRUM comes along, and its all about this “lets do the old thing, in the new way” excitement. Re-inventing the craft for a new age. I thought devops was that.
But that initial view is fading away.
System Administrators have a weird legacy. Very very few of us set out from our youth into the craft. More common are stories, fables really, about the Physics PhD who got more enjoyment out of the computers than they did the calculations they ran on them. From various technical paths, they came and feel in love with the tool rather than the task it was used for.
Now look at our books and our talk. We view our worth in our expertise in various technologies. The Linux expert, the Storage expert, the HP-UX expert, whatever. And knowing all the surrounding technologies and how they fit together, such as DB2 installation, performance and backup on AIX using IBM POWER systems. Up to date on the latest LTO advances, an answer for every little hickup of NetBackup. I mean, look at the most fundamental books for aspiring SA’s, they tell you how to learn the various UNIX OS’s, install software, manage hardware, back shit up, and use scripts to glue things where necessary.
These devops guys I’m meeting don’t fit that criteria. Or even that legacy. They are something new.
Everyone is quick to tell you that “devops” is not a role, nor a title, nor a job description. Its a culture. To which I say, enthusiastically, right! Except… the devops guys I keep meeting don’t look, act, or sound like systems administrators. I was most aware of this at LISA this year. Looking around at my fellow administrators, I didn’t get a “devops” vibe from the vast majority. The uber-nerds around the LOPSA table? No devops vibe.
The people that I met that seems to “get” the devops culture are devs. But not that kind… You see, they profess years in administration or ops, but they more excited by JSON and AMQP than they are by the next release of their favorite OS or chip architecture. These people no longer see computers as autonomous entities, rather they see the protocols to link them. They are more excited by NoSQL databases that can house metrics than they are in the commands and interfaces they can get them from. Spending time learning every little feature of an OS isn’t exciting, but optimizing inter-system communication is.
What I’m driving at is that the most interesting artifact of the “devops” movement is that its bringing a new generation of Systems Engineers out of the closet. They are devs in an ops world. Not dev for dev sake… not ops for ops sake… but rather dev for the sake of ops.
These people say its about bridging the divide, tearing down the wall, etc. But I don’t buy it. The most astute among them will recognize that you will still need “experts” on either side of that wall, who might not “do much” (which is one hell of an understatement) but do it very well. So, this idea that “devops isn’t a title” and yet “we still need specialists” seem to logically conflict.
It is therefore my assertion that “devops” isn’t a job title, but Systems Engineer is, and when you look at the folks really exclaiming the joys of it, they really seem to me to be Systems Engineers, or fanboys, who are spreading their joy of having found their new nitch in the universe.
Were not done yet……. let me explain why I think this is important.
As the good news of “devops” spreads it first enlightens, then brings excitement, then dread. If your one of those “specialists”, you can easily feel that your now out-dated. Consider that there is now pride within the devops elite that CIO’s are now talking about having a “devops strategy”. Some even suggest a (I’m paraphrasing) “evolve or die” scenario for operations teams. If your a sysadmin who uses Borne or Korn shell instead of Ruby, look out! I don’t think that’s fair, nor do I think its true for all. Instead, it all makes more sense when you see it as three camps instead of two, with a the culture over the three… that is, applications developers (traditional “dev”), system administrators (traditional “ops”), with a new role in the middle of Systems Engineers that helps glue the camps together. Some of your Systems Engineers will emerge from the dev side, some from the ops side, always having hidden their secret urges to do both. And, as with any emergent role, many will aspire to it but simply not be cut out for it.
Devops is a culture. Yes. Devs get it, Ops get it… but this new generation of Systems Engineers have it in their hearts, and you can see it in them. They are special… not better, just different. Like all of God’s creatures.
Thank you, Ben for this great post!
I agree that we must absolutely avoid that this third role/camp as you call it, becomes the third reich, feeling über knowledgeable on both sides. Sure, it brings value to the table, but the way you describe it, it brings only technical/engineering value.
The cultural shift we are hoping for is for people collaborating in whatever role they are. This can effectively be done without having people that fall into the category of the new middle systems engineers. Plain old devs and plain old ops working together can work beautifully. After all it’s not a technical problem; in a lot of companies, it’s a huge (mis)communications problem.
Again, thank you for your insights, they are very valuable!
Excellent post, Ben. Really.
You had me at the definition of systems engineering and the photo of Gene Kranz.
While there’s a special type of Utopia where all devs and ops have ‘full stack’ awareness and perspectives, the breadth of our field is so large that domain expertise will dictate that this glue you speak of is mandatory.
It’s excellent to be part of a field that’s seeing this transition.
Nice work.
The who concept of devop is bizzare, as to me it just seems part of the job of being an op. I guess it must come from the MSCE guys who can barely click their way out of a menu. I guess power shell has just got them back with the program again.
Arh well, lets watch this go round again…
I couldn’t agree more with this article. Here I was just thinking I was just a developer who was also a sysadmin. I never felt that I was a true sys-admin and yet I am not a classic developer since much of my time is spent in the ops world. Devops explains my niche perfectly!
System (note the singular) engineer is a superset of a system administrator in skill and experience. He understands CMMI and ITIL. And knows how to put that knowledge to good use when designing systems, architecting applications, or developing networks.
It’s that simple. Really it is.
Where a sysadmin will want to glue several systems (usually a small number) with scripts, it is the system engineer’s job to stay him and say:
“wait! What are we doing here? If you hack that script together, it’ll work, but the next guy won’t have a clue where you stuck it, or how it works! Let’s sit down, analyze what it is you are trying to solve, and solve it for the entire network. Let’s write a formal specification of what you’re trying to do, give it a requirement ID, and track the progress. Let me develop the code for you, test it in our lab, and PACKAGE IT, so that you may have it deployed for you automatically on the entire network. Furthermore, let’s think about how we are going to have this discrete component tested automatically, and let’s make sure we have documentation for you, on how to use it, an operating manual.”
Basically, system engineer’s job is to automate and encapsulate stuff and think about the concept of running systems and networks on a very large scale, because if you can solve it for a very large number of systems, you have solved it for a single system. The management concept will scale.
Ich bin ein devops!
Or at least I want to be one. It feels good to see I’m not alone in coming to operations from a systems perspective (in the larger sense).
@kangcool: I’m not sure it’s a matter of platform or OS. I’ve worked with plenty of sysadmins (good ones at that) who cared more about hardware than writing unit tests. I need that expertise on my team as I have to admit that it leaves me cold. I’m the one they call when they need a new package or when mercurial misbehaves. The part about being more excited by metrics and NoSQL databases rang very true to me.
Great post Ben.
Here’s another point to consider:
Dev and Ops used to be very different tasks using different tools and language. In the SaaS world everything is melting together. For example: Ruby is so successful because it has Ops in mind. Software companies are writing tools for SaaS companies that help deploy, monitor, manage… well you know… operations kinda stuff.
As you say: developers think the tools are cool because everything is written with integration in mind. Linking things together is a challenge they can solve with their tools and their language.
So in my view DevOps is a discipline – it’s an awareness of the developer that, especially in the SaaS world, development and operations are inseparable.
Hey Ben,
Awesome post; I’ve personally been trying to aspire to the DevOps “ideal” – more development with an operations focus, though I come at it from a wholly ops perspective. I just had an awesome programming session with my CEO, which has renewed my faith in having devs and ops unite together in some way.
Also, you were at LISA? I totally missed you this year, I was at OpsCamp at LISA, attended a few talks, then headed off to RubyConf. Maybe I’ll see you next year?
I enjoyed the article. I think DevOps is a little more than just the reemergence of systems engineering, though that is a lot of it. The team I managed here for six years was the “Web Systems” team and all the people on it were “Web Systems Engineers.” We aligned strongly with the business and worked with both the developer teams and also system administration specialists in tech-silo teams in Infrastructure (UNIX, Network, DBA, etc.) exactly in the way you describe. So obviously when DevOps came around we were “right there” and interested because of the similarities between what we’d been doing and the new hotness. But I think that what we’ve gotten out of DevOps gave us a little more than our systems engineering as usual – one part was moving to Agile for operations, for sure. Also, it’s the discipline of closer collaboration between the halves – using “dev” techniques to solve ops problems and vice versa. We seeded a team of both devs and ops to create some new products, and though I’m using all my systems engineering stuff, it is more than that as well.
From one point of view, there is some conflation of DevOps with ‘big picture,’ and from that point of view, sure, architect > engineer > administrator > operator/support tech and each higher grade is more of a big picture person. But I think DevOps is useful even at the lower levels, and within “pure dev” and “pure ops” teams. One of the things we learned about being systems engineers is if the pure-ops (or pure-dev, but that was slightly less common) teams you were working with didn’t give a shit about collaboration or business outcome – things went badly. So yes, in a large organization you probably do end up having specialists glued with generalist groups, but there are still profound alignment and tooling issues that can help that go well or poorly.
Almost every young people have one or more NFL jerseys sale . It is not only a way to show the pride for the favorite teams, but also comfortable when you are exercising. NFL jerseys sale are the best choice for wearing in summers. The authentic jerseys are so expensive that not any average football fan can afford. Then, you can get it online for bargain.
Great article. Thank you for your provided information.
[...] list goes on for where we should be looking for influences, and other folks have noticed this as well. As our field recognizes the value of taking a “systems” (the C. West Churchman [...]