Steve Jobs was many things to many people.  He liberated the innovations of XEROX PARC and brought them to the masses through yet more innovation.  He brought style together with technology and forged an unending bond between the two.  Apple has revolutionized the world multiple times over 3 decades under his leadership, and proved that it was his leadership that made the difference when the company almost failed after his leaving.   On and on….

… but for me, I will also remember him most of all, for NeXT.

Rest in peace Steve.  We’ll miss you.  Thanks for leaving the world with a dominant OS that is UNIX based on laptops that “just work”.

The following is the video of Conan O’Brien delivering the 2011 Dartmouth College Commencement Address.  Watch it, if your time is short, skip to 19:40.

This is an incredibly insightful point that Conan makes: the difference between who you want to be and who you actually are is what truly makes you special and unique.

This profoundly resonates with me. I have many heros and I’m incredibly blessed to know many of them. I want to be that fine combination of dreamer and engineer that are Bryan Cantrill, Jeff Bonwick, and Carsten Haitzler. I want to be the operational manager that is John Allspaw. I want to have the contagious enthusiasm of Adam Jacob or the conviction and pragmatism of Theo Schlossnagle. I want the wisdom of Russel Ackoff, the eloquence and curiosity of James Burke, the theological intensity of Charles Haddon Spurgeon… I could go on and on. I have a great many role models.

But the thing is, I’m not any of those people and as hard as I try I never will be… and I have tried. Often our role models are in tension with one another. However, like our taste in music, which are similarly in conflict (I like Megadeth, and Stevie Wonder, and The Bird and the Bee, and Rachmaninoff, etc), those preference themselves say something unique about who we are.

I find great personal comfort in what Conan says because it matches up perfectly with what I’ve found in my lifes journey thus far but not been able to articulate. I share it with you because I know a great many other SysAdmin’s who similar have felt “If I can just be as good as that guy I will have made it!” Except, you never become that guy and when you at last reach that level your horizons have expanded and you realize that your course isn’t the same as his/hers.

What is profound is the point that it is the difference, not the similarity, that is truly important. That is what really makes you unique. That is the place from where you can truly contribute. Therefore, don’t ever hold yourself back or silence yourself until you reach some imaginary peak. Don’t be afraid to code just because you think your code is lousy, don’t be afraid to write just because you don’t think your as smart as someone else, etc, etc. One thing I’ve found about my heros that I’ve met, while I look at where they are as my destination, they themselves are on their own journey and can’t understand how anyone else would put so much value on their position because they themselves aren’t yet where they wish to be.

As Emerson put it: “Life is a journey, not a destination.” Whatever you have now is what someone else is wishing to aspire to, so never despair and never hold yourself back.

I’m preparing to leave on a long overdue vacation. When I return I hope to be back on-form, and to finally get this blog rolling again. I’ll spare you the explanations as to why I’ve been so silent (for now)… but, every day I find more and more reflecting back to this video:

If you can see the embedded video: Click Here

This monologue of James Burke, from the last episode of his original Connections series has had the single most significant impact on my life of anything I’ve ever read, heard, or seen, and it explains so much of the world around us today.

I’ve always felt uneasy about the pairing of “computer” with “science”. There are two reasons. First, “science” is conventionally realized as the observation of nature; since computers are man made the rules we may derive from observation and analysis may not be applicable when we build a better or different computer. The second is that “computer science” in industry rarely seems to exhibit any of the distinguishing marks of “science”.

Perhaps you have had similar thoughts… but like so many things that flutter in and out of our heads on a daily basis never got around to digging deeper on the issue. I therefore thought I’d pass along some interesting research/discussion on the subject for your personal edification.

The first is The Cruelty of Really Teaching Computer Science (PDF) by E. W. Dijkstra written in 1988. The paper is hand-written and a joy to read. Its a bit long, but reads quickly. When reading it I found my self nodding a lot and feeling a joy that someone of such immense talent and honor has throughly given thought to these things.

The second is a piece in Wired from 2007 in which Bruce Sterling summarizes an ACM interview with Peter Denning: Computer Science. Is it Really a Science, and What’s It a Science About? in which the subject is address head on. Peter here discusses his project to produce a set of solid principles of computer science, to put these questions to bed. The results of that project are found on the site: The Great Principles of Computing project

These two resources should help you significantly along your way. There are many more, but these two stand out to me. And, because I think they are so interesting and important, I shall not do you the disservice of any further description here.

It does, hopefully, raise in your mind a question on you can answer about yourself: “Am I an engineer or a technician?”

As I pointed to in a recent post, an Engineer is “the discipline, art and profession of acquiring and applying scientific, mathematical, economic, social, and practical knowledge to design and build structures, machines, devices, systems, materials and processes that safely realize solutions to the needs of society.”

A technician is “a worker in a field of technology who is proficient in the relevant skills and techniques, with a relatively practical understanding of the theoretical principles.” So says Wikipedia. The entry continues with a useful example: “For example, although audio technicians are not as learned in acoustics as acoustical engineers, they are more proficient in operating sound equipment, and they will likely know more about acoustics than other studio personnel such as performers.”

Dijkstra argues that we do ourselves a disservice by preferring the term “bug” (which seems somehow passive and inevitable) instead of the more accurate “error” (which is active and direct). In the same way, I think we do ourselves a disservice by blurring the lines between “engineer” and “technician”.

Lets take a little journey through a thought and see where it shall take us.

Let us first consider “devops”. It is a cultural movement which codifies various characteristics, including the zen like interweaving of development and operations, better integration of “IT” with the business objectives as a whole, infrastructure as code, agile (or lean depending on who you speak with) operations, etc, etc. Alright. As I’ve stated in the past, its all the things ITIL claimed to be but somehow more hip. Alright, capture that in your minds eye and then move it aside.

Next consider the title “Systems Engineer”. It is a very ambiguous title to be sure, because it is cross discipline. Borrowing from WIkipedia: “Systems engineering is an interdisciplinary field of engineering that focuses on how complex engineering projects should be designed and managed.” OK… now, lets break it down into its consituent parts. An engineer is, Wikipedia says: An engineer is a professional practitioner of engineering, concerned with applying scientific knowledge, mathematics and ingenuity to develop solutions for technical problems.” Alright, and “system” comes from Greek literally meaning: “whole compounded of several parts or members, system”. Great, now put it back together… a System Engineer is one who solves technical problems using science and math to to make diverse components act harmoniously together as one. Great, put that aside.

As a geek, I have two pictures of the classical “systems engineer”. The first is a guy at NASA building the lunar lander, pocket protector and slide rule in hand, he’s making sure he can cram everything he needs into a landing module while still fitting on top of a rocket and capable of re-entry into the atmosphere. This guy has to think outside the box, consider every detail and yet be flexible enough to change the plan, like splitting the lander into two pieces, one which stays on the lunar surface and the other that blasts back to earth, etc. The other guy is a “kernel programmer”, I put that in quotes because maybe they might build compilers or drivers or libraries, but these folks know their ASM and don’t think FORTH is out of style yet. Most of these guys are up there in years, work at stodgy companies, and don’t have much of a sense of humor. Externally these guys don’t seem much different… ones at AT&T or IBM, the other NASA, but the same image:

Lets now consider OS’s and computers. On the whole, OS’s aren’t changing. Is Linux fundamentally that different than other UNIX’s? We’re not seeing radical shifts in what we use. Similarly, in the chip space we’re all now comfortable with the reality that 5GHz CPU’s aren’t going to pop up any time soon (at least, not in general use) but we’re going to get more and more 2.xGhz cores. 2, 4, now 6, etc. Even your parents little Dell now has 2 or more CPU’s, soon your kids will have a laptop with 6 or more… and they won’t know or care. The point is that we’ve hit a maturity point in which the building blocks are pretty solid, but the way in which we use them is now more interesting. The old image of the Systems Engineer feels, well, old.

OK… lets now knit this together.

When I first came in contact with the “Devops” movement I thought it was about better systems administration, a maturing of the craft. It was, I thought, more about Ops and less about dev. Dev had their day in the sun with Agile… now sysadmins were getting theirs. Remember, if you can, when Agile came on the scene and how giddy developers got with eXtreme Programming, then Pragmatic Programmer books are everywhere, and then SCRUM comes along, and its all about this “lets do the old thing, in the new way” excitement. Re-inventing the craft for a new age. I thought devops was that.

But that initial view is fading away.

System Administrators have a weird legacy. Very very few of us set out from our youth into the craft. More common are stories, fables really, about the Physics PhD who got more enjoyment out of the computers than they did the calculations they ran on them. From various technical paths, they came and feel in love with the tool rather than the task it was used for.

Now look at our books and our talk. We view our worth in our expertise in various technologies. The Linux expert, the Storage expert, the HP-UX expert, whatever. And knowing all the surrounding technologies and how they fit together, such as DB2 installation, performance and backup on AIX using IBM POWER systems. Up to date on the latest LTO advances, an answer for every little hickup of NetBackup. I mean, look at the most fundamental books for aspiring SA’s, they tell you how to learn the various UNIX OS’s, install software, manage hardware, back shit up, and use scripts to glue things where necessary.

These devops guys I’m meeting don’t fit that criteria. Or even that legacy. They are something new.

Everyone is quick to tell you that “devops” is not a role, nor a title, nor a job description. Its a culture. To which I say, enthusiastically, right! Except… the devops guys I keep meeting don’t look, act, or sound like systems administrators. I was most aware of this at LISA this year. Looking around at my fellow administrators, I didn’t get a “devops” vibe from the vast majority. The uber-nerds around the LOPSA table? No devops vibe.

The people that I met that seems to “get” the devops culture are devs. But not that kind… You see, they profess years in administration or ops, but they more excited by JSON and AMQP than they are by the next release of their favorite OS or chip architecture. These people no longer see computers as autonomous entities, rather they see the protocols to link them. They are more excited by NoSQL databases that can house metrics than they are in the commands and interfaces they can get them from. Spending time learning every little feature of an OS isn’t exciting, but optimizing inter-system communication is.

What I’m driving at is that the most interesting artifact of the “devops” movement is that its bringing a new generation of Systems Engineers out of the closet. They are devs in an ops world. Not dev for dev sake… not ops for ops sake… but rather dev for the sake of ops.

These people say its about bridging the divide, tearing down the wall, etc. But I don’t buy it. The most astute among them will recognize that you will still need “experts” on either side of that wall, who might not “do much” (which is one hell of an understatement) but do it very well. So, this idea that “devops isn’t a title” and yet “we still need specialists” seem to logically conflict.

It is therefore my assertion that “devops” isn’t a job title, but Systems Engineer is, and when you look at the folks really exclaiming the joys of it, they really seem to me to be Systems Engineers, or fanboys, who are spreading their joy of having found their new nitch in the universe.

Were not done yet……. let me explain why I think this is important.

As the good news of “devops” spreads it first enlightens, then brings excitement, then dread. If your one of those “specialists”, you can easily feel that your now out-dated. Consider that there is now pride within the devops elite that CIO’s are now talking about having a “devops strategy”. Some even suggest a (I’m paraphrasing) “evolve or die” scenario for operations teams. If your a sysadmin who uses Borne or Korn shell instead of Ruby, look out! I don’t think that’s fair, nor do I think its true for all. Instead, it all makes more sense when you see it as three camps instead of two, with a the culture over the three… that is, applications developers (traditional “dev”), system administrators (traditional “ops”), with a new role in the middle of Systems Engineers that helps glue the camps together. Some of your Systems Engineers will emerge from the dev side, some from the ops side, always having hidden their secret urges to do both. And, as with any emergent role, many will aspire to it but simply not be cut out for it.

Devops is a culture. Yes. Devs get it, Ops get it… but this new generation of Systems Engineers have it in their hearts, and you can see it in them. They are special… not better, just different. Like all of God’s creatures.

I’ve been watching and silently participating in the industry trend known as “DevOps” for some time now. I’ve dragged my feet on writing about it for reasons that I won’t go into now, but this blog post, which made it to slashdot, has pulled me out of my shell. Ted Dziuba picks a fight by saying:

“This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them – Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don’t overthink things, and arrive at simple, scalable solutions.”

So Ted wants to sensationalize his amazing realization that basic UNIX tools are actually useful by dumping on Devops. Fantastic.

Slashdot includes a link to Patrick Debois What Is This Devops Thing, Anyway? which Ted does not. While Patrick is rightly credited as kicking this movement off and dubbing the term DevOps, his blog entry describing it is vague, at best. Ted summed it up, wrongly, as “SA’s start writing unit tests and other things to help developers warm up to them”. Wrong wrong wrong.

So lets break this all down and get to some reality.

DevOps is a buzzword which identifies a new cultural evolution in systems administration. It comes from the WebOps world where the divide between programmers who write the software (“dev”) and sysadmins who deploy and manage the software (“ops”) is paper thin. Patrick put a buzzword and some conferences around the growing trend of sharing methodology across this boundary.

So, the first manifestation of “devops” is a world in which developers learn from operations and operations learns from developers. The result is ops teams using Hudson for continuous integration testing, unit testing everything produced in ops just like dev does, always pushing trunk, allowing features to be enabled or disabled in software without having to rollback code, etc. So, in this world, the two groups act much more uniformly in process and tools.

But, the second manifestation is more interesting. Administrators outside the webops space are being caught up in the momentum of the movement and listening to the same talks but hearing different things. This is currently mostly affecting SMB and startups which have been, for the last 3-4 years, greatly influenced by the webops shops.

To this latter group, Devops is really “Next Generation ITIL; ITIL for the masses”. They hear the Devops guys say things like “Operations needs to serve the business, not dwell in a cave of geeky solitude” or “Operations needs to measure everything and have release process, change management, source control, etc.” What a lot of people are hearing is “ITIL is kool again”. Not just ITILv3, but also CobiT, CMMI, ISO20K, ISO27K, etc. Because the concepts that really strike a cord with many systems administration teams are clearly defined in ITILv3. For those who find ITIL too big to chew off, the book Visable Ops by Tripwire founder Gene Kim, et al, has become a cult classic in how to implement ITIL in a realistic and sensible approach.

The reason this push for ITIL is different than the first, is that now ITIL is being adapted and pushed from the bottom up, as opposed to the first wave which was forced from the top down. Engineers, not executives, are thinking “Ya, change management sounds good, but how do I do that?” and they find ITILv3 already spells it out. The effect is that ITIL is now what it was intended to be, a collection of best practice which can be used as a reference by organizations to augment and improve existing practices. The first wave was bastardized by CIO’s chucking out the baby with the bathwater so they could look busy. But I digress…

The other side is that we’re seeing a real push in devops for a unified toolchain. Check out the Devops Toolchain Project. The open source administration tools are getting really solid. This toolchain breaks down into the areas of monitoring, provisioning, configuration management, and command and control. The movement to standardize as an industry on CFengine, Puppet or Chef, has cemented the second brick of the foundation. Improving command and control solutions (ControlTier, Nanite, mcollective, etc) are getting very close. The important result is an industry push to standardize on a similar set of tools thus providing re-use between organizations.

Finally, sprinkle Agile in to taste. A little SCRUM or Kanban, a touch of Jira, a little Git Hub, a pinch of manifesto, shake vigorously to combine.

Whats the catalyst though? Why now? None of this is really new, so why do we suddenly seem to be getting our act together as an industry?

Cloud. We’ve all been told for years that the number of machines we’re managing is going to explode and, as is did, our methods would have to change. What we saw was a slow ramp and with it we slowly adapted traditional methods to increase numbers of resources. However the traction that cloud computing has brought finally set off the explosion of systems.

Let me use an age old analogy you’ll all be familiar with. If you have to do something less than 10 times, big deal, do it manually. If you have to something more than like 10 times you might as well spend the time writing a script to automate it. But if you have to do it 100 times, then you really have to get your act together and build a reasonably robust solution.

Cloud pushed us as an industry from 10 to 100. Writing piles of ad hoc scripts won’t cut it anymore because you don’t know what your environment will look like in a year. It might be bare metal boxes, it may be virtualized in house using VMware, or it may be a cloud solution based on Joyent, AWS, and Rackspace. Whatever it is, you have to change the way you approach old problems of provisioning, configuring, monitoring, and managing systems.

So “systems administration” starts maturing into “operations”. Process is more important. Policy is more important. Standardization on generic tools like Puppet or Chef is more important. IT can no longer be an illusive crack squad of hackers that do things in a way that ensures both l337-ness and job security. IT has to be scalable as an organization and is more and more becoming a real revenue generating part of the company that has to actually act like part of the company by answering to the board like any other business unit.

All these things are colliding together. And at just the right time in history, Patrick Debois comes along and says “Devops” and proposes rethinking the way we do operations. No one really cares what Patrick says after that because so many of us have been rethinking all these things and just didn’t have a word to put to it. “Devops” is the word.

Words have power. Thanks to “Devops” people are now coming together, sharing together, learning from each other. We’re not alone it seems, which means we all don’t have to re-invent the wheel.

Web 2.0 might have been a crappy buzzword, but it marked a cultural change in what users expect from their web experience. Web 2.0 was a cultural evolution. Cloud might have been a crappy buzzword, but it marked a cultural evolution in how we architect our infrastructure, from a physical model to a virtual model. In the same way, “Devops” is the cultural evolution of what is expected from IT and Systems Administration. Its been a long time coming and all the pieces are now aligning and as an industry we’re finally maturing. Focus on the “culture of change” or you’ll just miss the whole point.