Within an ops team you should have 3 primary types of governance enablers: controls, policies and processes. A control is a guiding principle, which is implemented as a one or more policies (which are just rules), which are in turn standardized in a set of procedures. Its important to have all 3, because controls are very vague, policies are often general and broad in nature, which means to provide consistent quality results we require prescriptive procedures. At Joyent we call these “Standard Operating Procedures” (SOP).

The whole point of an SOP is to produce consistent results regardless of who’s using it. That means that all SOP’s need to be in a similar, familiar, and easy to follow format that is suitable to anyone who may need to use it. That, therefore, means that to get those consistent results there can be no room for ambiguity, it must be explicit and convey any necessary context along with it. Ambiguity is the mortal enemy of consistency. Case in point, if you’ve ever been asked to recompile software with a large number of configure flags, if your unable to determine which flags were used in the past you’ll go cold with anxiety over whether or not your building it properly. When you go back and ask who it was built in the past someone might say “Don’t you know how to compile software?” and the answer is likely going to be “Yes I do, but I don’t know how YOU compile software.” Whats important is that the person implementing a procedure be given all the information and context necessary to understand, and if necessary, interpret the information as appropriate for the given situation.

The first key to better SOP’s is to provide a template for others to follow. Without a standard template each author will write the procedure in their own unique style. Some people will write you a book, others will just paste some lines from their terminal into a code block. The template therefore must enforce a certain flow that ensures we include all the needed information but in a concise and complete way. Plus, we want SOP creators to focus entirely on writing the content, not debating the format.

Here is the template I use for Joyent Operations SOPs (in Confluence markup):

* Author:  {page-info:created-user}  created at: {page-info:created-date}
* Version: 1
* Revisions: {page-info:current-version}
* Reviewed by: (User @ date)
* Time to implement: 1hr
* Products this applies to: (SKU1)

{toc}

h1. Description & Scope

h1. Prerequisites

* Root access to node
* [SOP-222: Something|SOP-222: Something]
* [SOP-224: Something else|SOP-224: Something else]

h1. Procedure

h3. Step 1: Do this

{noformat}
Example
{noformat}

h3. Step 2: Do that

h3. ...

h1. Procedure Validation

# Login and verify external connectivity (ping google.com)
# curl zone IP address, page returns
# etc.

h1. Notes/Jira Examples

* [http://confluence.atlassian.com/display/DOC/JIRA+Issues+Macro]
* [http://confluence.atlassian.com/display/DOC/JIRA+Portlet+Macro]
* [https://studio.plugins.atlassian.com/browse/CONFJIRA-154]

Lets step through the above template.

All SOPs must be numbered for easy reference. Even the template itself is SOP-000. The SOP title is in the form: “SOP-102 Creating LDAP Users”, for instance.

The top of the SOP is full of metadata. The author, creation date, major version number and number of revisions made and products (or projects or whatever) that this SOP applies to. You’ll notice 2 other fields: “Reviewed By” and “Time to implement”. These are perhaps the most important of all. After an SOP is created it must be reviewed by someone else in the group, preferably with as little knowledge of the subject as possible. They should read and follow the SOP as written, starting a timer when the begin and stopping the timer when they are complete… it is that stopwatch time which becomes the “time to implement”. This is extremely important, the time estimate for implementation by the author will be way too short because they know what they are doing, the time it takes a complete n00b will be more useful and truthful.

Moving on through the template, “Description and Scope” are where we provide context. What are we talking about, what does it entail at a high level, what does this impact, etc. We want to include as much information as possible to set the stage for the procedure that follows. Then we include a bulleted list of “Prerequisites”. The single most common part of any procedure that gets skimped on is the prerequisites and they are also generally the most time consuming.

The meat of the SOP is the procedure itself. I strongly believe these must be in a “Step 1… Step 2… Step 3″ format; it must be easy and intuitive to follow and in some cases may be used as a checklist during sensitive procedures. Its important that these truly start at the beginning and go to the end. “Step 1: Login to server X” may be overly simplistic but necessary for clarity if multiple machines are involved. I also like to have the final step be “Done” to make it clear that you have reached the end.

Just as important as the procedure is the “Validation Steps”… to ensure a quality job we must not only preform the proceedure but validate it in one or more ways to ensure it was really done right. This has the added side effect of giving the person doing the work the satisfaction that it was done properly and they didn’t screw something up along the way.

Lastly is a place to include external links as appropriate. If possible I like to link in tickets (we use Jira) which have relied on the SOP before, so that if by chance there is some confusion they can find examples of the work being done in the past.

An optional section that I’ve used before is a “Rationale”. In this section you would include notes on why you chose to implement the procedure in the way that you did. This allows for continuous improvement of the SOP. In most cases there are many ways to solve a problem, conveying why you chose the method you did will help you hone the procedure in the future while learning from the past. Without it your likely to have regression or duplication crop up.

This is the model that we’ve used at Joyent for several years and it has stood the test of time. I believe it to be a very solid standard for writing SOP’s and sharing knowledge within the organization and avoiding any one single person becoming a constraint. If you have refinements or a better method, I’d love to learn about it.

Last week I was given the incredible opportunity to not only speak at LISA but to deliver the opening keynote.  I hadn’t expected to even go, but when I learned the topic was DevOps I made a last minute plea on the eve of the submission deadline for a slot to deliver a talk I was calling “The 60 Minute MBA”, a history of Operations Management.  My hope is that I could get some obscure timeslot so a handful of people could geek out with me on Operations Management and LEAN and how it is helping to fuel and direct a lot of the DevOps thinking out there.  To my great shock I was told I was given the keynote slot… frankly, something I didn’t want for fear of the stress associated with it, but Tom felt I should step up and that I’d do great.

I haven’t blogged much in the last year and when I have its on topics you probably wouldn’t expect from a “Solaris blogger”.  I’ve held back most of what I want to talk about and only let the cream rise to the top.  My already frantic reading backlog only intensified as I was trying to pack as much into my talk as possible and ensure I was accurate.  Everything I read, watched, attended or did was reshaping my talk and I essentially spent 6 months “on stage” in my mind.   The problem I really had was that I had maybe 6 hours of content that I needed to condense into a 1 hour slot, hitting the essentials but not diluting its potency.  And, of course, I’m still learning every day.  Only 2 weeks prior to my talk did I finally hammer out a rough slide deck and I then had to keep pushing it around into something moderately cohesive.  Trying to find ways to address wisdom, systems thinking, agile, lean, TPS, OM and OR, and tie all this back to DevOps was a challenge.

To make things more challenging, Tamarah’s (my wife, seen above) due date for our 5th child is the 14th of Dec and the talk was to happen at 9:30AM Eastern time, which is 6:30AM Pacific and I’m not a morning person.  So… all things considered, I did pretty well, but you will notice in my talk that I was a little slower than I normally would be.  The upshot, however, was that I didn’t ramble much which kept me on my time marks.

What was interesting to me was what different people walked away with. Some people really keyed in on the value chain and asking “Why?”. Others wanted to rediscover ITIL because it was the first time they had heard it didn’t suck. Others got interested in operations management and LEAN, something they’d heard of but didn’t know where to start learning more. Others keyed on the collaboration of devops and bringing teams together. There was, I think, something for everyone and I didn’t hear any negative feedback on that talk beyond some people liking some parts and not caring about others… and it was designed that way.

Two things I want to note for viewers. First, when I said “by men I mean the human race”, I should have better explained that I think of “men” in a JRR Tolkien sense, the “race of man”. Secondly, at the very end I bagged on Sun TechPubs… I didn’t really explain myself and someone took offense to it. The fault was not on Sun’s writers, but rather on the engineering managers who wouldn’t permit writers the access to engineering they needed, so TechPubs was left to figure it out themselves. The fault was squarely on the engineering managers, NOT on the writers. Given the circumstances they have always turned out amazing documentation and I have nothing negative to say about the writers (as I noted in my answer, I wanted to be one at one time).

Anyway.  The following is the keynote, the slides can be found here.

I referenced a lot of books, and may have asked for the list of books, so here it is.

Please note! I do not profit from any of this in any way, I’m not getting a book kick back or whatever.  My only source of income is my Joyent salary.

The Essential Books you should read to put DevOps, ITIL/ITSM, LEAN and Operations Management into perspective and educate yourself for the future:

1. The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps
2. Any Operations Management textbook
3. Web Operations: Keeping the Data On Time
4. Lean IT: Enabling and Sustaining Your Lean Transformation

The Advanced Books you can read to dig behind the ideas, this is my “Best Of” list:

1. My Philosophy of Industry & Moving Forward Henry Ford
2. Today and Tomorow Henry Ford
3. The Principles of Scientific Management Taylor
4. The Toyota Production System Ohno
5. Out of Crisis Deming
6. The New Economics Deming
7. Management Challenges for the 21st Century Drucker
8. The Goal Goldratt
9. Critical Chain Goldratt
10. Creating the Corporate Future Ackoff
11. Future Shock Toffler

One book mentioned in my talk that I do not own, nor have I read, is Lean Startup by Eric Ries, which is based largely on The Four Steps to the Epiphany a book I did buy at the MIT Press bookstore after my keynote. “Lean Startup” is popular, but all he’s really doing is applying LEAN concepts and Agile methodologies to the startup. There are hundreds of “Lean XYZ” books. I am personally interested in the real deal, not books about other books. “LEAN IT” is my one exception because it can be a big time saver and I feel it gives proper credit to the history and sources of the ideas it espouses.

Finally, rather than give you a “fire hose” list of everything, I’ll simply include a picture of what I feel is a very complete libary on these various topics.  The handful of books missing from these shelves are PDFs on my iPad such as the official  “ITILv3 2011 Update”, several books on Engineering Systems, etc.  Click the image to see it high-res.

8 All things are full of weariness;
a man cannot utter it;
the eye is not satisfied with seeing,
nor the ear filled with hearing.
9 What has been is what will be,
and what has been done is what will be done,
and there is nothing new under the sun.
10 Is there a thing of which it is said,
“See, this is new”?
It has been already
in the ages before us.
11 There is no remembrance of former things,
nor will there be any remembrance
of later things  yet to be
among those who come after.

 

Ever been irritated by the subtle but constant reference by Agile and DevOps people to manufacturing?  You may not even realize they are doing it, but you’ll hear reference to a book called “The Goal”, quotes from Deming, analogies to factories, etc.  In many conference talks I could feel that there was some larger body of knowledge that speakers were alluding to, but not fully describing.  What was this secret knowledge?  Last year I finally stumbled upon the answer and I’ve been consumed by it ever since… long time readers of my blog will note a considerable change in tone and subject since Dec of last year.

This secret body of knowledge that is all around you, but not directly named is “Operations Management” (OM).

Classically, it is said that a company is made up of 3 primary organizations divisions: Finance, Marketing (which includes Sales), and Operations.  Finance handles the books and internal resources, Marketing brings the market to the company and sells its products to that market, and Operations is the part of the company that does what your company does.  This is an overly simplistic model, but it makes a complex organization easier to grok.  If you run a hot dog stand, “operations” refers to ordering hot dog stuff, making hot dogs, serving customers, etc.  If you make cars, “operations” refers to the factory floor managing supply chain, operating the assembly line, and delivering cars to dealers.  If you run a web site, “operations” refers to the developers and sysadmins who make the product, run it, etc.  So again, the model breaks down to bean counters, sellers, and makers/doers.

Have you ever thought about getting an MBA?  I have.  Except, when I looked at the curriculum my eyes somehow danced right over OM, because I didn’t know what I was looking for.  Now I know.  You can examine the OM departments at Harvard Business School and MIT Sloan.  As with so many things today, the first step to knowledge is knowing what to look for, if you don’t know what its called you can search until your blue in the face and find nothing of real value.

My journey really took off when I found, at Church of all places, a donated text book entitled Fundamentals of Operations Management (4e).  “WOW!” I though, “that what I’ve been looking for!”  One look at the table of contents and I knew I’d stumbled onto the illusive body of knowledge I’d sought for so long:

1. Introduction to Operations Management
2. Operations Strategy: Defining How Firms Compete
3. New Product and Service Development, and Process Selection
4. Project Management
5. The Role of Technology in Operations
6. Process Measurement and Analysis
7. Financial Analysis in Operations Management
8. Quality Management
9. Quality Control Tools for Improving Processes
10. Facility Decisions: Location and Capacity
11. Facility Decisions: Layouts
12. Forecasting
13. Human Resource Issues in Operations Management
14. Work Performance: Measurement
15. Waiting Line Management
16. Waiting Line Theory
17. Scheduling
18. Supply Chain Management
19. Just-in-Time Systems
20. Aggregate Planning
21. Inventory Systems for Independent Demand
22. Inventory Systems for Dependent Demand

Jack pot!  If more than half of those chapters don’t seem pertinent to IT departments, then you’ve never tried to manage one.  The focus may be slightly different, but the core issues, problem domains, and related disciples are essentially identical.  This explains why so many “experts” are making reference to OM, knowingly or unknowingly, because in manufacturing they dealt with the same problems, in essence, we have in IT.  The Web companies (Twitter, Facebook, Flikr/Etsy, etc) are the ones leading the charge because more than traditional IT organizations, they really do look like the factory floor producing a single line of products.

So now… now I know what questions to ask.  And ask I did.  This opened up a whole new world to me that was right under my nose.  The Toyota Production System (TPS) which became known in the US as “Lean”… W. Edwards Deming and Total Quality Management (TQM)… ISO-9001…. the undertones of ITIL, CobiT, ISO-27001, and Agile…. it all came together and made sense for the first time.

This sent me into an epic journey as I sought out book after book after book by the cornerstone individuals of OM, because they all wrote books that formed the modern body of knowledge.  I now own all of Henry Ford’s books, Shigeo Shingo’s books, Taiichi Ohno’s books, W. Edward Deming’s Books, Walter Shewhart’s book, Fredrick Winslow Taylor’s book, Ludwig von Bertalanffy’s books, Peter Drucker’s books, and on and on and on.  I couldn’t stop buying and reading these texts that describe the world we find ourselves in today, shaped by the work they did so long ago.  All these points in my head started to be connected, one by one, and a fabric of knowledge appeared.

Friends, the point is this: there is nothing new under the sun.  Things change, evolve, and morph, sure, but the principles are not new.  If they were, we wouldn’t look back at Plato and Aristotle as wise today, much of what they debated 2400 years ago is still as pertinent today.  So it is with Agile and DevOps, the core principles have been well explored and addressed in the last century of manufacturing as part of Operations Management.  We only need adapt that knowledge, and the “experts” are doing exactly that.

Consider an example.  As a consequence of the innovations Ohno was introducing at Toyota in building the Toyota Production Systems (TPS, aka Lean), and in particular that of Kanban (the basis of Just-in-Time production, which is pull rather than push based production), he needed a way to speed up the “changeover time” (setup time) of large pressing machines.  These machines contain “die” which press sheet metal into, say, a car door.  The changeover time could be as much as 6 hours… that means, when you decide to stop making part A and want to make part B, you have to shut down for 6 hours to setup the machine for the new part before starting production again.  The way this was typically handled was to simply make a shitload of parts to build up a big inventory so that you reduced the likelyhood of needing to do another setup.  They were after local efficiency (what the “Theory of Constraints” calls local optima) at all costs.  This mass production method wasn’t going to work in Ohno’s new just-in-time world, the idea of stamping out only 20 parts and then changing to create another was completely idiotic.  At least, it was until he put Shigeo Shingo on the job.  It too Shingo years to make it happen, but ultimately he created a method know as “Single Minute Exchange of Dies” (SMED).  With his method you can change dies in less than 10 minutes (single-digit minutes, not 60 seconds).  This was the breakthrough that Ohno needed to make Kanban really work… and work it did.  With out SMED, a technology approach, to compliment Ohno’s other methods (Kanban, 5S, 5W, Andon, Muda, etc) Toyota just wouldn’t have been the industrial revolutionary that they became.

Now, why the hell am I telling you all that?  Look at what cloud did to IT.  Just like Kanban, Cloud came along and showed us that our setup times are way too long, and changeover from one type of setup to another was awful.  Configuration Management (CFengine, Chef, Puppet, etc) are the SMED of our industry.  Same problems, same needs, different solutions, but similar approaches.  There is no reason for us to re-invent all the wheels, alot of these issues are solved problems, if you just know where to look and what questions to ask, and have an open mind.

If you are like me and have been looking for something, but you know not what, go find yourself a book on Operations Management and get your journey started.  You’ll have a massive head start over all your peers who won’t figure this out for another couple years (just as others already got a head start over us).