Devops != Sysadmin

WARNING: This post may be misleading. What I call “devops” below is not what the canonical description of the term seems to be. Please make sure to read comments after this post to get a better picture.

In the comments to my Rise of Devops post from earlier this year, Damon Edwards asked an interesting question:

Why do you consider DevOps a standalone discipline? Isn't what yo described just the characteristics of state of the art systems administration?

In other words, is devops a standalone thing or is it just a fancy new name for a sysadmin. I couldn’t answer the question back then, but now, having thought about it for a bit, I am at least in a position to offer an opinion.

Most IT organizations consist of 2 groups of people: developers and non-developers. Whether these two groups work together or hate each others’ guts, doesn’t really matter for the purposes of this post. Non-developers group is traditionally subdivided into facilities, systems, network, storage, applications and security. In smaller teams, an individual could be a part of several functions from this list (people call it “wearing many ops hats”); in bigger teams each function could be its own department; in extra large teams, each function may even have its own director. Developers write and maintain code that powers the business, non-developers assure whatever is needed for the code above to work properly, is there and works as it should.

But there is another hidden taxonomy for non-developers group. If we look at it with the focus on individual activities instead of on people’s job descriptions, you will see this group consisting of:

  • activities around hardware purchased from vendors
  • activities around software purchased from vendors (excluding software delivered as a service)
  • activities around software and infrastructure delivered from vendors as a service
  • activities around open-source and/or freely downloadable software
  • activities around software developed in-house

I believe that the first 2 types of activities are mostly SYSTEMS ADMINISTRATION - common theme here is MANAGE or ADAPT. One needs to perform day-to-day tasks like user account management and troubleshooting; identify, prepare, test, perform and validate changes; analyze utilization and plan capacity expansion. Because items under management in this category are purchased from vendors, the expectation is that vendor provides everything necessary to run their stuff successfully, and sysadmin’s main goal here is to adapt it to run optimally in this company’s environment. Analogy here is you mold something into shape. To excel in this category, one needs a lot of knowledge about a particular system, its gotchas and peculiarities. Such knowledge is usually obtained through experience through trial and error, vendor trainings and user group meetings. One can totally do this without software development background.

The other 3 types of activities, on the other hand, are mostly DEVOPS (infrastructure as code) - in contrast to MANAGE and ADAPT, main themes here are CREATE or ASSEMBLE. It’s not taking something that already exists in some form and molding it into the shape you need - it’s taking a lot of standalone pieces and getting them to work together (like building with Lego). There is no way this can be done successfully at scale without software development background - devops is dominated by developers who turned into ops, or ops who have been doing a lot of serious scripting all along.

Interestingly, startups are more likely to lean towards hiring devops-minded engineers, because these days a lot of startups don’t need to manage their own hardware anymore and often can build their products on software stacks which do not include any proprietary software (that would need to be purchased from vendors). There might not be a need in sysadmin at startups anymore, but it doesn’t mean the sysadmin skills are no longer needed anywhere else in IT - just look at all the big IT companies on NASDAQ and NYSE selling big iron or complex software systems.

It’s also worth noting that separation could be not as clear-cut - for example, controlling hardware devices programmatically, using published or unpublished APIs or telnet & expect voodoo, definitely fits under devops.

Therefore, in my opinion, devops is indeed a standalone discipline, but it doesn’t mean that a given person’s role in organization could be devops. We are all engineers, and difference is what kind of activities one performs during a given day - when you perform a firmware upgrade on a piece of hardware, it’s sysadmin; when you put together a cluster management system for your nodes in EC2 - it’s devops.

P.S. Yes, I know that what I keep calling “devops” is not about what others call “devops” - culture of collaboration between developers and ops. I just don’t know what else to call this.

Categories: devops |

Comments (13)

Damon Edwards // 07 Oct 2010

Definitely an interesting and accurate taxonomy of the kind of work that goes on inside IT organizations... but I'm still very much against the idea of "DevOps" being a job description.

I'm going to steal from John Allspaw who once summed it up nicely in less than 140 characters as to why this is a bad path to go down. Calling "DevOps" an IT job title is having a job title on the business side of the house called "Communication and Collaboration".

I do agree with your statement that there needs to be a name for the role you pointed out in your bullets. I've seen that role in many orgs and they call it "Systems Engineer".

Making something a job title has the unfortunate effect of causing others care less about it. "Oh I don't have to worry about how my actions impact Dev (or Ops).. that's the job of that DevOps guy in the other cube!"

-Damon

[...] Read more at http://somic.org/2010/10/06/devops-and-sysadmin/ [...]

Daniel Kushner // 07 Oct 2010

Dmitriy - kudos on the post. Nice to see an angle based on activity rather than role. Another parameter to look at is the management of application tiers. While you have expert DBAs, Storage Gurus, and the such, Devops needs cross disciplinary knowledge since application release deployments are usually across these, coming with their dependencies in a holistic release.

+Daniel http://www.noliosoft.com

Dmitriy // 07 Oct 2010

@Damon

Communication and collaboration theme is important only in a certain scenario - when code developed in-house is involved. Please see Devops - Solution to a Problem, Not a Cure for All Ills.

Why would a sysadmin working on a complex SAP or Oracle Applications deployment need to collaborate with developers? What kind of feedback could developers provide to him in this case that will help him do it better?

Why would a sysadmin setting up a global printer solution need to collaborate with developer?

Why would a sysadmin setting up a global AD infra need to collaborate with developer?

Why would a sysadmin working on internal DNS need to collaborate with developer?

John Allspaw // 07 Oct 2010

Sorry, I wholeheartedly disagree with devops != sysadmin. The separation of expectations and responsibilities in-house and out-of-house software is a non-starter for any successful web operation, and likely for some enterprise operations as well.

I'm seeing something in your comments that lead me to think there's a separation you're trying to get at with what I'd call sysadmin responsibilities within the "Corporate IT" responsibilities (Active Directory, Printing, etc.) and what I'd call "Production Operations", or the sysadmins working on the site.

I'll say that communication and collaboration theme is important only in a certain scenario - the business.

Collaboration on topics/projects/infra/code that don't have shared concerns critical to the business isn't a requirement of what's known as the DevOps culture.

Collaboration and communication on topics that *are* at the shared boundaries between dev and ops do have high value. But I don't think that has anything to do with titles or where the application comes from.

(For example, MySQL wasn't developed in-house, but I'll argue that MySQL is a very critical shared concern at Wikipedia, Twitter, Facebook, Flickr, Etsy, and Google)

Dmitriy // 07 Oct 2010

@John

Good point. In-house/out-of-house separation is indeed tricky - I did have it in drafts of this post but before publication realized same thing you are saying - that's why I went with "software purchased from vendors."

I am trying to extrapolate ongoing devops discussion to all of IT, not just web companies. "Production IT operations" at an airline or a retail chain or a hospital or a bank or an insurance company are very different from "production IT operations" where one's business is one's web site.

And it is this bigger world that includes non-web companies where I think "devops != sysadmin" stands.

I agree with you that in web companies "devops != sysadmin" is not a true statement or at least should not be a true statement.

Ernest Mueller // 07 Oct 2010

Actually, I think what you are describing is what we in house have called "application administrators" for a long time in our enterprise.

Like ten years ago, our IT Infrastructure group was composed of all tech-silo "sysadmin" groups. Unix, Network, Windows, Notes, DBA, etc. Then, starting with our Web site, we realized we needed a team of specialists to do Web systems and operations. We called them the "Web Admins", and they sat "between" the sysadmins and the developers; the systems teams did OS builds and rack-and-jack and hardware procurement and data center and storage and network and that stuff; the Web Systems team did app servers and load balancers and app deploy and was the single point of contact for operations (and eventually this branched into Web performance/availability/security engineering). Over time people realized the success of that model and other "app admin" groups were formed to handle internal applications.

Now these teams, in our org, are the ones getting interested in DevOps first, because they are the ones usually working more with the developers on specific projects. I was our Web Admin manager for a long time and I'm our chief DevOps evangelist. But I'm not sure if that's not just a historical artifact. If the systems teams had been more successful in collaborating with developers in a DevOps manner, would the 'app admin' teams have been as necessary?

Anyway, I sometimes bridle at the term "Ops" in DevOps because it does cover a wide variety of "non programming staff" - in a large shop like ours, that's specialized out to sysadmins vs app admins, but even in the app admins you have "systems engineers" vs "operations staff" vs "architects" vs "release"... Of course, many development shops have all the same distinctions (service programmer, UI programmer, architect, etc.)

In the true sense of the "DevOps" word, it just means fostering agile collaboration between ALL technical stakeholders to achieve working products - using "Ops" to indicate syadmins, app admins, network folks, security folks, whatever. I think what you are pointing out is just a meaningful distinction inside Ops that isn't a tech silo distinction.

Damon Edwards // 08 Oct 2010

Dmitriy,

I actually agree with pretty much all of what you are saying. My only (respectful) objection is to how you are using the word "DevOps".

DevOps is a cultural and professional movement focused on eliminating the bottlenecks and inefficiencies that plague IT organizations and lead to both business and morale problems. It's a conceptual umbrella the loosely binds a wide variety of concepts together under a shared problem statement/goal.

DevOps is not a person, job function, or title. This isn't just my blowhard opinion... it's from talking to a lot of people at conferences and meetups (from Velocity to OSCON to the DevOps Days series to regional DevOps meetups) and interviews I've done for blogs and podcasts.

Using "DevOps" as a job title is like using "Agile" as a job title for a developer who uses Agile methods. At the best it's confusing, at the worst it is misleading.

If you need a better word to describe someone who "creates and assembles" applications, why not pick a new word(s) rather than bend "DevOps" to fit? I've seen it called "Systems Engineering". Ernest calls it "App Admins". There are many more out there already in use for years.

You even said yourself in your post: "P.S. Yes, I know that what I keep calling "devops" is not about what others call "devops" - culture of collaboration between developers and ops. I just don't know what else to call this."

Overloading a word like DevOps with your own different definition is only going to create more problems than it is going to solve.

Of course this is your blog and you can tell me to shove off if you like... but perhaps this discussion would make a lot more progress if you changed the premiss from what DevOps is/isn't to a focus on coming up with consistent naming for those roles you described.

Thanks for letting me put in my $0.02.

-Damon

Ernest Mueller // 08 Oct 2010

@Damon - although just to be clear, I think "app admin" and "systems engineer" are different axes - we split our "app admin" group into systems engineers and production support (we called the latter "operations"). I think a better taxonomy of Ops would help a lot of these discussions.

Dmitriy // 08 Oct 2010

@Damon

I value your opinion and appreciate the time you took to write it up.

As a prominent member of the community, you certainly have more authority to define the term, and yes, I agree that overloading its meaning is not going to lead to anything good.

I added a warning on top of the post which I hope will prompt people to read the entire conversation so that each visitor can form their opinion based on a bigger picture, not just my individual opinion.

Again - thanks for you helping us all understand it all better.

Cheers, Dmitriy

Tom // 15 Oct 2010

Dmitriy

as some others already mentioned this is not new stuff at all. As an example than I was at Compuserve in 2000 we called this operations platform engineers providing an ongoing, stable and more or less state of the art platform to all those business and online developers. Providing and permanently enhancing that operational platform was the only choice to survive on the operations side. Every supporting task was either automated or outsourced to others within IT department.

We - aicooma - support and integrate that type of platform management and engineering since 10 years and tons of media- and online companies I know within Europe still use our idea of platform engineering, even as interface and supporter for both insourced and outsourced solutions.

So what's really new on that? The idea to make a methodology? Not, even newer ITIL versions or dialects adress that topic.

Using agility for operations? No, eg. we use KanBan for years to migrate DataCenters and keep on developing the ops platform.

I think to adress that idea is still a very important task ,thanks for that. But it isn't that new.

Tom

turbidostato // 17 Oct 2010

Terminology demotion.

You said: * activities around hardware purchased from vendors * activities around software purchased from vendors (excluding software delivered as a service) [...] are mostly SYSTEMS ADMINISTRATION - common theme here is MANAGE or ADAPT

* activities around software and infrastructure delivered from vendors as a service * activities around open-source and/or freely downloadable software * activities around software developed in-house [...] are mostly DEVOPS (infrastructure as code) - in contrast to MANAGE and ADAPT, main themes here are CREATE or ASSEMBLE.

In contrast to Damow Edwards' "Why do you consider DevOps a standalone discipline? Isn't what yo described just the characteristics of state of the art systems administration?"

From my point of view you falled in the Scotsman's fallacy. What you say it's within "devops" realm is nothing but what has been expected from a sysadmin for the last, what? 40 years? You call it by a different name because there has been a "demotion" of the sysadmin concept as it becomes obvious from what you think to be "true" sysadmin tasks.

Any greyback sysadmin here will tell you that what you tell "sysadmin" is not work for system *administarator* at all but the tasks for a system *operator*, his lessen brother (and usually the kind of PYF work that will open the door to become a sysadmin in the future).

What you tell to be "devops" tasks are, and have been from the begining, the "proper" sysadmin tasks you give to your senior staff. In other words: if it's managing something others did automated or layed down for you, it's not systems administration, it's system operation. It's system administration when it is *you* the one making the thinking of and the automation for others to execute it.

It would be too long to explain where this demotion comes from but I think it suffices now to say that some well known company from Redmon has a lot to do with it.

Dmitriy // 17 Oct 2010

Strongly disagree.

I know very well what the differences between sysadmins and operators are, and nowhere in this post I talk about operators.

I have no idea where in the post you saw a link to a certain company based in Redmond either.