DevOps is not DevOps
Wednesday, June 21, 2017
I learned DevOps the hard way about five years ago when I inherited some infrastructure and software that had zero automation and monitoring. Since then, I’ve run into many teams that called themselves as “DevOps team”. My most recent encounter was with a team that wanted me to review their “DevOps” org structure and portfolio. By the end of that conversation, that team thankfully changed their name into something more appropriate. They were just building a set of automation tools for other teams to use.
During this time, I’ve also run into some really good engineering teams that simply didn’t want to have anything to do with DevOps. Their rationale was that operations work is menial, and they didn’t want to waste their precious team deploying, upgrading, monitoring and performing other upkeep tasks. Let someone else deal with those was their answer. They just wanted to keep coding.
I’ve also come across another kind of engineering teams that originally embraced the DevOps mindset, built some automation, practiced some level of CI and CD tasks, but failed to keep the automation up to date with their software architecture. I recently met someone from one such team, who was hoping that another team would take over some of their core components and their operational responsibilities. His team was not able to keep up with those responsibilities. The other team in his case was a centralized team that manages an odd assortment of tools and systems that nobody else wants to manage.
Another team that started with a similar DevOps mindset ended up hiring a “DevOps” manager and creating a “DevOps team” to take care of operations as they could not improve their automation and operational processes to meet the increased scale.
I feel sorry for such teams. They were all missing an important element called feedback, and potentially paying a price. However, lecturing such teams that they must own and take care of their software themselves may not be effective.
A better approach may be to probe about how they intend to maintain feedback between all aspects of software lifecycle such as architecting, coding, shipping, running, and operating their software so that they can be agile. Ask about how long any existing feedback processes take.
Here is why. DevOps is nothing but a culture of maintaining short and healthy feedback loops between all lifecycle activities related to creating and running software and systems.
The shorter the feedback loop between these activities is, the more agile the team can be. Agile teams have a better chance to learn quickly from mistakes. Their architectures can evolve fast. They are less likely to pick architectures or processes that do not support reasonably quick feedback.
When feedback loops through these activities are long, mistakes take longer to observe. Corrective actions take even longer to yield results. More important, hypotheses remain untested for months. Ground can shift in the interim.
Below is an example that shows potential interdependendencies between architecture and code, automated delivery, monitoring and SLA management, chaos experiments, and cost forecasting and optimization.
Each of these functions impact one or more of the others. For instance, analysis of the architecture tells how to test for failures, while a chaos experiment may reveal some inherent weaknesses in the architecture or the absence of some resiliency patterns in the code. Similarly, the architecture chosen may be prohibitively expensive to run when fully adopted, and thus may require changes to the architecture itself.
You therefore want the arrows between various functions take as little time as possible, so that you can apply lessons learned and be able to iterate. A team that practices all these feedback loops is more likely to succeed than a team that does not. Org structures and team names don’t matter as long as there are healthy and short feedback loops between various functions.