The Rise of DevOps

If you are in IT, you probably noticed that most of the industry’s technical buzz lately has been centered around one of three huge areas - cloud computing, nosql and devops. Unlike Web 2.0 or Social Web, which are about content generation and content consumption models on the Internet, these three are actually about how software systems are built and operated - it is “engineering” vs “product.”

DevOps is on the rise as a newly re-defined standalone discipline, as evidenced by increased number of good articles about it around the blogosphere. In this post, I am going to take a stab at outlining what DevOps means to me.

I’ve got some devops cred. Before joining my current employer where my role morphed over time away from devops, for over 2 years I had been at Orbitz.com where I was in a group in charge of monitoring and automating a hugely distributed multi-datacenter custom airfare search application, running on many hundreds of machines, with several times as many separate entities and processes that needed to be coordinated, restarted, tweaked and so on (our group was in charge of everything above hardware, OS and basic network services such as connectivity, DNS and DHCP). Before that I had had various sysadmin roles, which all involved a large degree of coding beyond the level of simple shell or perl scripts.

To me, devops is a distinct discipline at the border between software engineering and ops, which focuses on developing software for the infrastructure on top of which end-user-facing software is running. It’s sometimes referred to as development of infrastructure software and includes release deployment. Devops has the following distinguishing characteristics.

1. Ability to write code beyond simple scripts

Obvious necessary condition.

2. Focus on stability and uptime

Stability and uptime in devops almost always trump features.

3. Extra focus on moving between states

In dev land, I have often observed situations when the end result of a particular feature was analyzed on its own merit, without taking into consideration how a system can be moved from its current state to its desired future state. Devops pays extra attention to this problematic area.

4. Different angle on business revenue

While developers usually work on things that are meant to increase or sustain business revenues, devops often work on things that are meant to prevent or reduce loss of business revenues. This is somewhat similar to defense vs offense in team sports. The key word is “balance.”

5. In devops, we are users of our own software

This is one of the most important distinctions. Unlike developers who create software to be used by someone else (internal customers, end users, site visitors, etc), devops is about developing software for internal needs. For example, you can certainly get sloppy in logging that error, but it’s you, not someone else, who’s going to suffer the consequences of having to waste extra time to find necessary information.

6. Architect, developer, tester, product manager, project manager - all in one.

My personal experience in devops is I/team get(s) an area of responsibility and it’s up to me/us to make it happen. Assigning priorities, figuring out dependencies, reacting to unexpected changes, managing resources - all of these functions are performed in devops by the same group of individuals.

7. Awareness of normal accidents

I have an entire blog post dedicated to this - check it out.

8. QA in production

Some tasks in devops can’t be adequately tested in smaller synthetic environments. Lack of scale, lack of unique hardware, lack of sufficient capacity in vendor’s test environment, lack of sufficient connectivity from test site to vendor’s systems - all could be factors. Phased deployment and other techniques designed to reduce the risk of a complete meltdown are (or should be) used extensively in such scenarios, but the truth is - from time to time in devops I had no other way but to actually run a test system in a live production environment.

9. Manual first, then automate

In my experience, a devops task is more likely to start out as something done manually at first, and automated later. In dev land, tasks rarely go through manual phase before being coded up and shipped in a release.

10. Almost always distributed or hyper-distributed

Conclusion

Devops is on the rise primarily due to realization that there is a big gap between developing end-user systems and bare-bones systems administration, in large part due to fast growth of IaaS cloud computing. Devops originated at places where a relatively few sysadmins were in charge of many hundreds or even thousands of hosts - where doing their job without automation was impossible. As time goes on, I expect devops will further solidify its role as a first-class citizen and make inroads into non-cloudy companies as well.

Categories: devops | infrastructure-development |

Comments (8)

Damon Edwards // 02 Mar 2010

Your list of DevOps characteristics is a great addition to the DevOps discussion. Thanks for the link to my "What is DevOps?" post.

Why do you consider DevOps a standalone discipline? Isn't what you are described just the characteristics of state of the art systems administration?

Wouldn't that be like saying someone is an "Agile" rather than just saying they are a developer who is working in an Agile development methodology?

-Damon http://dev2ops.org

ian // 02 Mar 2010

awesome article! I spend a lot of my time focusing on things like this -- there really is a HUGE difference between developers who have to develop for end-user needs and those who have to write more than simple scripts to manage their systems and use the tools THEMSELVES -- one of the systems I had to develop was a custom deployment script -- options such as puppet/chef can't keep up with our speed of change and were meant for larger distributed systems whereas i just needed to deploy diff. projects to a couple of different hosts

Dmitriy // 02 Mar 2010

@Damon

This is a good question, and I am afraid I don't have a good answer yet.

It's clear to me that dev and devops are not the same thing, because the skillset and experience required to excel in each, are different.

However, I do see your point that devops could be just _a_ way to do systems administration, in which case it would not be a standalone discipline.

John Allspaw // 02 Mar 2010

Good work, Dmitriy. :)

Damon Edwards // 02 Mar 2010

I'm not so sure I'm right either :)

I came to my conclusion that DevOps is not a job title from two observations I've seen repeatedly during my company's consulting engagements.

1. When you look at all of the roles you have to touch to make the promise of DevOps a reality, it goes far beyond just an updating of how systems administrators work. Dev, QA, Project Managers, etc... eventually all need to get on board and play a part in making the end-to-end DevOps process a success (although systems administrators will bear the bulk of the heavy lifting)

2. Many companies already have a "man in the middle" group who is responsible for getting changes from development deployed into production. Often this is called the "systems engineering" group. Most of those companies fail when it comes to DevOps because development and the rest of the operations team leave it all up to them ("it's your responsibility, not ours... so don't tell us to work in a way that seems sub-optimal for our group")

[...] the job are the people that are skilled in development and systems administration (these are the DevOps people everyone is talking about). We’ve developed a lot of processes and tools to do our job [...]

[...] The Rise of DevOpsLists 10 characteristics, not all technical. [...]

[...] The Rise of DevOps [...]