The Rise of New Operations

It has been 6 years since I wrote a blog post titled The Rise of Devops. Many things have changed during this time and I realized a re-evaluation could be interesting.

Today, in 2016, here is where I think we are.

1. Operations main focus is now scalability

In the past, our primary purpose in life was to build and babysit production. Today operations teams focus on scale. For some it could be traffic related (number of concurrent sessions, number of users, size of the dataset). For others it could be ability to move between states safely and at high pace (for example, fintech where high stakes make consumer web approaches to operations too risky).

Automation is a necessary condition of scalability, and if you want everything to be automated, you must have devops as your goal.

2. Operations team is now a foundation of your platform team

Engineering teams exist in order to create product. With many teams in a company, there is often a need to have a dedicated team whose job is to ensure productivity growth - code reuse, implementation patterns, client libraries, core APIs. This is what I call a platform team.

Historically, this platform team would partner with opertations but today we are already at the point where operations is a part of the platform team. I have another post on this topic.

3. No more dedicated operations teams at early stage companies

This is NoOps and I realize that many readers will have strong objections here but I believe it’s a new reality. Look at any early stage company - there are rarely pure operations roles among co-founders or early employees. At that stage, it’s a luxury that only a few can afford - when you are iterating very quickly, you don’t worry yet about scaling your service for large crowds that may never come - you care about features and market fit. People may wear many hats, and some will still do a lot of operations but as an add-on, not as their main function.

A pure operations role in the form it used to exist at young tech companies is now extinct.

4. More and more enterprise workloads require webscale

This is an emerging trend, where traditional enterprises start selling products and services directly to consumers and quickly discover that their operations chops in webscale world are lacking.

This is especially obvious to me in part because I live outside of the Bay Area.

5. What got us to our current state?

I think 2 main factors - IaaS cloud (along with AWS unstoppable quest to standardize everything that is not code itself, by offering it as a service) and operations people’s widespread adoption of software engineering productivity techniques (in other words, operations people becoming better developers and moving up from scripts).

6. Where are we headed within next 6 years?

I am pretty confident that in another 6 years a lot more truly large scale applications and services will be successfully running without a dedicated operations team. Where all engineering teams have at least one customer-facing product service running in production, operations team no longer exists as a standalone part of the engineering organization (which does not mean that operations expertise or tech ops knowledge or best practices become not needed).

I think it’s the future.

Categories: devops |

On Employees Investing In Their Startups

Like many others in tech, last week I saw a story in New York Times about a startup whose financial exit did not quite meet its employees’ expectations. There is also a good follow-up by @StartupLJackson, which I recommend.

There are several things that I think are important to understand in the context of this situation.

Side note and a disclaimer. I am not trying to convince you to make any investment decision one way or another - my goal is to merely illustrate some aspects of investment analysis thought process that I strongly feel you should be considering. Times article clearly tells me not everybody pays attention to these, which is unfortunate.

No matter what anybody says, the final decision is yours and yours alone - in tech ops we say “you own your availability,” in the investment world “you own your investment decisions” carries the same weight.

On private company valuations

I frequently come across comparisons of privately held company valuations with those of publicly traded companies. While I understand that sometimes people use such charts to add color to the point they are making in which case these comparisons are just illustractions roughly approximating reality, I suggest you be careful taking them literally.

Stock of a publicly traded company has many technical characteristics that make it significantly more transparent to a trader or investor than stock of a startup. There are SEC filings that are public information (interestingly, as an investor you don’t need to read them all - the fact that they are published and there is a legal framework that ensures the filings are very likely to be true, mean that information will be priced in the shares when you are going to buy), there is daily trading volume supported by liquidity, there are short sellers keeping stock price honest, often there are publicly traded derivatives - all these factors combined make it easier to understand and value (assign price to) public stock.

All of these things are either missing or very limited for private companies. In extreme case, I could negotiate purchase of a single share of a startup directly with management at a sky-high valuation. The price I will pay is a result of only 2 opinions about how much a company is worth now - mine and management’s, and valuation from last transaction could be used as basis for future stock sales.

Also, keep in mind overall stock structure and share classes. With publicly traded company, stock structure is simpler (fewer classes) and better documented - if a most sophisticated investor in the world and I own a share of AAPL, we hold absolutely the same thing with equal rights. (Yes, I am aware of dual-class share structures in public companies, etc).

In private companies, multiple share classes could exist with very different terms which significantly impact how much each share of each class is worth depending on a lot of variables (think liquidation preferences, for example).

The fewer classes company stock has, the easier it is to come up with market cap number using simple math. For a startup, calculating market cap is harder (sometimes significantly) - if a company issues preferred stock during last round of financing, it doesn’t mean you can automatically take that price and apply it to common stock.

Sum up: comparing valuations of publicly traded companies and private companies is often (or at least frequently) the same as comparing apples to oranges. If you want to know what you are doing when investing, you should analyze such comparisons with great deal of caution.

On investing hard cash in the startup where you work

Like with any investment decision, it’s impossible to give a general recommendation in this case without knowing company details, your personal financial situation and your appetite for risk. But nevertheless, the thought process you should follow when deciding whether to invest or not is universal.

Mathematically expected value of your equity grant most often starts out as strictly positive - there is a set of outcomes when it’s 0 and there is a set of outcomes when it could be worth something, so overall it’s greater than 0. In other words, you may not win but you will not lose. If you add to your position and pay cash for it, understand that this statement is no longer true - there is now a chance you can lose real money.

When investing in your startup, just like when you make any other investment, you should consider several aspects:

  • expected return on the investment relative to other investments in your portfolio and other asset classes available to you (is the risk justified? is this the best use of your investment dollars on risk adjusted basis?)
  • opporunity cost (what else could you do with the money you are about to invest if you decided not to go ahead with this investment?)
  • total loss (how severely will you be impacted if you lost all of your investment?)
  • diversification (if this investment goes through, what will it do to your portfolio diversification?)
  • time horizon (how is the expected time horizon of this investment affects timings of your future financial goals?)
  • liquidity risk (it may be very hard to convert your investment to cash when you need it - do you have enough cushion in your portfolio?)

Furthermore, contrary to popular belief that an investment decision is either “yes, invest” or “no, don’t invest,” in reality you can not decide whether to invest or not without knowing the price you will pay and terms of your investment. As a general rule, almost always there exists a price at which even the worst company in the world could be a reasonable investment.

When you are buying shares of a publicly trading company, you usually first determine whether you like the company and want to own its shares, then figure out a good price (or set of prices) at which you’d be willing to buy, and then execute your plan.

Unfortunately, coming up with the price you should pay for shares of your privately held startup is the hardest task of them all. It’s very possible that you are at a significant information disadvantage relative to sophisticated investors (which in case of tech startups are angels and VCs), founders and management. You have to find ways to compensate for it.

Sum up: Investing in anything (startups, stock market, real estate, bonds, gold, tulips, etc) without having a very clear idea why you are doing it and how various plausible outcomes could affect your finances, is a slippery slope. Not only are you deciding whether to invest, you also need to pay attention to price you will need to pay and terms associated with your investment.

On JOBS Act

This section is US specific. One of the parts of JOBS Act focuses on establishing a legal framework for startups to raise money from individuals who are not sophisticated investors.

I have been skeptical of this idea for a long time, and the Times article once again reaffirmed my skepticism. It’s a free country, and the fact that something is legal does not necessarily mean that you should do it.

I remain super convinced that it’s nearly impossible for an outside unsophisticated investor to overcome information disadvantage when investing in equity of a tech startup. And if I am a tag-along (attach my investment dollars to those of a sophisticated investor), I don’t see why founders would ever want to give me terms they are giving angels who are bringing their Rolodex, expertise and experience.

I do see how unsophisticated investors could loan money to startups (it would be a bond, which is senior to all equity investors - bond holders are paid first), but I am not sure whether it will work for founders (and existing angels).

Sum up: In my opinion, NY Times article is a very strong empirical argument why unsophisticated individual investors should not make equity investments in startups, even if it becomes legal under JOBS Act.

Categories: startup |

Unexpected Downside of a SaaS

It just occurred to me today that despite all the positive things that SaaS companies continue to deliver, a lot of them have an interesting weak spot.

In order to be successful, a SaaS must deliver superior product to what I would otherwise be able to build myself, plus deliver it at a price that I’d be willing to pay. Since price of the service is a factor in sales, it’s better to price service at a reasonable level. This in turn drives a need for SaaS companies to pursue economies of scale - it’s much better for them to build solutions that can be sold to many customers vs. build a customized solution for every customer.

As a result, one of the things many SaaS companies would do is they would put every customer’s data into the same data store of some sort, and would ensure data confidentiality through their service.

And here lies the rub. By doing this, they end up with much bigger datasets than an original problem domain called for. Bigger datasets are harder to work with - they require more optimization, more security, more planning, more of everything. Sometimes the size of SaaS dataset even prevents them from building functionality that I would be able to easily build had I needed to build it only for myself, on my small dataset.

This was inspired by a realization that one of the SaaS services I am using, lacks a certain API which would be trivial for me to implement had I had all of the SaaS functionality under my own control but only for my data.

Could bigger not always be better in this context? Does this line of thought affect technical architecture of SaaS data stores? Should it?

Categories: software-engineering |

5 Pitfalls of Increased Use of Statistics in Tech Ops

Our field is currently undergoing a seismic change towards becoming more and more quantitative. While in the past a chart was viewed by many as state of the art, charts won’t surprise anyone today. In fact, we now have systems that can produce any number of charts at varying time scales with just a few clicks.

On our way towards data-driven technical operations where runtime data are collected to be analyzed by machines instead of humans, we will inevitably start seeing statistics play bigger and more prominent role in our data analysis systems.

As someone who is very interested in statistics, I see the following five potential pitfalls that are waiting for us as the use of statistics becomes more widespread in techops. I will order them by decreasing severity (this, of course, is very subjective).

Applicability of current estimates to the future

Statistics is about estimating parameters of entire population (see some terminology) based on a sample of observations. It’s very easy to take one of your backend services, calculate 95th percentile of its response time during top 4 usage hours on each of the recent 8 Wednesdays, calculate some of this sample’s statistics and then say next Wednesday will be somewhat like this with such and such error.

The problem here is that we currently take for granted the fact that past 8 Wednesdays are a good predictor of what’s going to happen next Wednesday.

Identification of good time boundaries for relevant samples

Let’s say you are studying performance characteristics of your database cluster. You take data for the past 2 months and analyze it as a single sample and come up with some results. What you neglected to account for, however, is that at certain time during these 2 months you upgraded disks on your database servers to much faster ones. Or you swapped out one library with another that led to a significent throughput increase.

I see this left and right all over the place. If your sample includes observations that are not similar, your sample is not good enough. See this for more details.

Data exploration vs detection of actionable alerts

There are two distinct use cases for statistics in tech ops. On one hand, you have looking to learn something from the data. It could be hypothesis testing or it could be trying to detect a trend or it could be predicting the future.

Another use case however which is completely distinct is trying to alert off of statistical data in near real time.

These 2 use cases do overlap a bit but in large part they are separate and a tool meant for one may not be a good fit for the other.

Dissemination of insight gleaned from data

Most people who are paying attention to what’s going on in the world get bombarded with thousands of statistics per day. These numbers often come from all sorts of “experts” and talking heads, who throw them in just to sound more knowledgeable. Looking for example? Any statement that includes “on average” without explaining how the sample was obtained.

It’s important to us because sooner or later we will have to share insight we gain from our data anlysis with others and we have to be aware that our audience’s understanding of statistics could be subpar. This is especially important if a person with whom you share results of your analysis is going to make a decision based on his or her interpretation of the results.

Low bar set by enterprise buyers

A lot of software vendors whose tools include elements of statistical analysis sell to enterprise. Unfortunately, state of the art at enterprise is so low that when a sales person says “standard deviation,” they immediately get enterprse buyers’ attention.

As a result, tools include statistics for all sorts of things and misuse them. Case in point - claiming anything about “66% of your sample lies within one standard deviation from the mean” without explicitly proving normality.

Conclusion

Obtaining valid results by applying statistics in tech ops is not going to be easy but it’s an inevitable next phase that should be embraced as a challenge.

For more, please see this post.

Categories: devops |

A Path to DevOps Through Platform/Product Alignment

The topic of DevOps in the enterprise has been discussed extensively. There are many people who have made a meaningful contribution here due to their level of expertise in both. There are others however who know only one side well and whose contributions are adding quite a bit of noise to the conversation. At times I feel that we are not moving forward on the subject but going around in circles.

I approach devops in enterprise as an end state. Aspirational end state that we are targeting to reach. Developers and operations sharing responsibility for the production environment, culture of collaboration, adoption of software development best practices in operations, heigtened awareness in development of run time and supportability issues - all these still stand. But the path to devops in the enterprise may end up being very different from that of a startup or a greenfield app.

(Side note: Devops is the same whether we are talking about a startup, a greenfield app or an enterprise - but paths to it differ. It's an important distinction.)

Devops is not easy in a typical enterprise. Discussing why this may be so is beyond the scope of this post. Instead, I want to focus on what to do about it.

I suspect that most enterprises could reach devops end state with fewer problems and sacrifices by introducing an intermediate step before devops. I couldn’t care less what it’s called but it can be described as platform/product alignment.

Take whatever org chart you have and identify all of technology. Draw a line not based on dev/ops but based on platform/product. Product is stuff you do for your customers, it implements your business logic. Platform is the rest, these teams focus on shared services and libraries as well as automating your runtime environments. Define what product delivers to platform (usually a set of binary artifacts like for example a WAR file to be deployed to a java container), sit back and enjoy watching your people's technical curiosity make things happen (or in other words - get out of the way of engineers).

The main trick here is that platform part of this alignment will consist of both developers and operations folks. Developers in these roles tend to be somewhat more inclined to adopt devops which helps this strategy a lot - instead of trying to convert everybody at once, you focus on those who are the easiest to convert to begin with.

One important note - under no circumstances should you draw any further lines within the platform team as it will kill the strategy. It’s extremely important to make sure your platform is one team, not two separate team reporting to a Senior Director of Platform. Everybody on the platform team is a Platform Engineer (with numerals like Platform Engineer IV if that’s how you roll), even if their current responsibilities are not the same. Remember that in ideal devops, a developer’s day is identical to operations folks’ day - writing code, planning code, peer reviewing code, deploying code, meetings (yes, I remember that we are still talking about the enterprise) and there is no need to have different titles for folks in dev and ops.

As platform part of your technology organization starts solving problems they are tasked to solve, they will inevitably stumble upon techniques and practices that constitute devops. Remember that devops is not an edict that was invented in a locked room and sent to the rest to follow - it’s a set of techniques and procedures that emerged in the trenches, from bottom up. When facing similar issues, smart people are more likely than not to effectively rediscover the principles that devops is all about.

And after platform reaches devops end state, I am confident that devops methodologies will gain critical mass in your organization sufficient to generate enough momentum to let your entire organization (product included) reach devops.

This idea is not rocket science. It’s simple and straightforward and I am sure there are a lot of organizations in the world who are aready doing it this way. It’s relatively painless and easy to implement, especially if you consider alternatives.

It’s good time to act now - please consider this reorg to set your team up for success in 2015.

Finally, please note that the title of this post says “a path,” not “the path.” I think it’s the easiest way to reach devops for a typical enterprise but it’s not the only one. The key is to realize that devops is not something you adopt - it's something you achieve.

Categories: devops |

Next Page