November 1, 2018

Building Better Software: All Roads Lead to DevSecFinOps

A brief annotated history of software development. DevSecFinOps is the logical next step in the evolutionary chain. It's the new right way to build great software.

Software is hard. There are many ways it can fail, all of which come down to one fundamental problem: the software doesn't do what it needs to do.

But here's the good news. Software is actually getting better! We're getting better at writing it. This isn't about the lines of code, trends aside they're the same as ever. It's about the team structures we're using to build our products.

And here's the better news. We know how to improve further. Most places just haven't thought ahead enough to put it into practice yet.

I'll give you a look into the future.

The answer is DevSecFinOps. I know that sounds silly, but give me a minute to justify it. I'll explain the problem, then do a quick skip through the evolution of practices.

Bad Software

Software that doesn't do what it needs to do.

"What it needs to do"? That's broadly defined, but basically it's everything a stakeholder needs the software to do. Functional and non-functional, explicit and implicit, external (user-facing) or internal.

This failure is either:

  • It doesn't meet its requirements, but no-one noticed.
  • It meets its requirements, but the requirements are wrong.

There's a million ways for these situations to arise. To paraphrase Tolstoy: "all good software is alike; all bad software is bad in its own way".

Here's some I've experienced personally:

  • We gave the users what they thought they wanted, but that didn't actually solve their problem.
  • We built something that would have solved the user problem! But by the time we got the software to the users, their problem had changed.
  • Our software used to solve a particular user problem well! But then we added a new feature, and didn't notice we subtly broke the old solution.
  • The system was fantastic! But then we got lots more users, and it didn't hold up under load.
  • Everyone loved our product, it was super-successful! But then we got hacked and a bunch of our user data got stolen.
  • Our software was fantastic, a beautiful system! Users loved it, we loved working on it. But it cost more to run than the value it provided to people.

You can think of many others, I'm sure.

Missing Metrics

The common thread here is that there were important metrics that we didn't consider carefully enough: user satisfaction, scaling, security, return on investment. This wasn't malicious or incompetent, just a blind spot in the requirements for our product.

To use technical jargon, it's a multi-variate optimization problem. We're trying to build software to do well in lots of ways at once.

And the fundamental problem is that most of those ways are opposed to each other. Security and UX are always at odds. Cost and scalability are similar. Future maintainability of the code vs getting to market quickly.

There are always trade-offs in software.

Developers: The Thin End of the Wedge

Developers end up deciding on most of these tradeoffs.

These decisions come up multiple times per day when building a system. We know that:

  • A more accurate decision leads to better software.
  • A quicker decision leads to us getting to market quicker.
  • For best results, the accuracy of the decision should be tested soon.
  • The longer it takes to be tested, the more expensive mistakes are to remedy.

This is where the structure of the product team has big effects. The goal is to make the decision both quickly and accurately. In practice even one of those is good.

Historically, teams did neither.

Dev (Waterfall)

I started building software professionally in the 90s. Where I worked, there was only one game in town for process - Waterfall. We sat in the middle of a big, complex process:

➡ Business Requirements
➡ Technical Requirements
➡ Business Analysis and Design
──────
Development
──────
➡ QA Testing
➡ Release Management
➡ Operations (Maintenance, Security, Finance)

Seven (at least) separate functions with walls between them. Functional teams tended to throw their output over the wall to the next team. A function couldn't start until everything before it was done.

The inputs and outputs were something like this:

(users => business requirements)
(business requirements => requirements)
(requirements => architecture + business rules)
──────
(requirements + architecture + business rules => software)
──────
(software => approval)
(software + approval => user-facing system)
(user-facing system => maintained, secure user-facing system)

20+ years later, the problems are obvious:

  • Development is a long way from users.
  • Their daily decisions are guided by Business Analysts, who are also a long way from users.
  • They're unlikely to have information about what users actually need.
  • This clearly leads to low-quality decisions.
  • The uncertainty also leads to slow decisions.
  • Once decisions are made, there's a lot of process and time before they can be tested.

Outcome:

  • Everything takes a long time.
  • Quality ends up being low.
  • Arguments and poor communication are common.

Bottom line:

  • Software is very expensive.

Dev (Agile)

Fewer walls. This is a huge move. Instead of a Development Team, the internal four functions are performed by a cross-disciplinary team, the Product Team:

➡ Business Strategy
──────
Research into User Problem or Need
UX and UI Design
Development
Functional and User Testing (CI)
➡ Release (Maybe? If CD)
──────
➡ IT Operations (Maintenance, Security, Finance)

Benefits:

  • There's overlap between all of the phases.
  • Developers talk to users regularly, participating in the research phase along with skilled researchers.
  • When there's a decision to be made, the devs have more information to work with.
  • From a team perspective, they're also closer to the researchers and designers, so they can get an answer quicker.

Problems:

  • There's still a solid wall between the Product Team and IT Operations.
  • This wall means the Product Team is unlikely to build something that's maintainable, secure, and cost-effective.

DevOps

This is a simple, organic shift from Agile Dev. Now the Release and Maintenance functions are part of the Product Team.

➡ Business Strategy
──────
Research into User Problem or Need
UX and UI Design
Development
Functional and User Testing (CI)
Release (CD)
Maintenance
──────
➡ IT Management (Security, Finance)

Benefits:

  • Much more visibility of the operational effect of decisions the devs make.
  • Fewer terrible tech choices (for instance: mature DevOps teams were unlikely to choose MongoDB, because they didn't want to operate it).
  • Quicker feedback loops because CI and CD end up being significantly faster.

Problems:

  • Still two functions handed off to IT Ops, so the old problems still exist there.
  • Security in this model is usually a "Department of No", brought in too late and then accused of getting in the way.

DevSecOps

Bringing Security into the team makes it stronger again.

➡ Business Strategy
──────
Research into User Problem or Need
UX and UI Design
Development
Functional and User Testing (CI)
Release (CD)
Maintenance
Security
──────
➡ IT Management (Finance)

Benefits:

  • It's much easier to fix security holes if you find out about them early.
  • Devs can be helped to a strong understanding of the security aspects of their work.

Problems:

  • We're still crossing our fingers and hoping that we hit our budget.

DevSecFinOps

Finally a holistic team building the product.

➡ Business Strategy
──────
Research into User Problem or Need
UX and UI Design
Development
Functional and User Testing (CI)
Release (CD)
Maintenance
Security
Finance
──────

Benefits:

  • People are exposed to the cost of their work.
  • They stop making arbitrary decisions about the trade-offs between cost and value.

Problems:

  • None! Everything is perfect.

But that's a huge team - where do I get the people?

It doesn't have to be a huge team.

I trust engineers. Give them the right information and support, and they'll usually make good choices. Even outside of the area they're trained in.

We saw this happen when Agile took off. With visibility of user problems, developers proved able to make good decisions around how to deliver value.

It continued to happen when DevOps came in. Knowing what the metrics were, developers did a good job of making their software more operable.

It's happening right now with DevSecOps. Teams and companies are upskilling their product teams on security, and getting more secure products out of that.

DevSecFinOps will definitely take the same pattern.

Here's the support product teams need:

  • The right metrics.
  • The right tools to stay on top of those metrics.
  • The right support to build knowledge they don't yet have.

The Brave New World

Moving from DevOps to DevSecFinOps is simple, though it isn't necessarily easy.

Start by giving your product teams the right metrics. Tell them how much their product is costing. Set up security guardrails and tell them when they hit one.

And then support them in improving these metrics.