The setting
Have you read Team Topologies? Continuous Delivery? You haven’t? Well, stop everything you’re doing right now, go get these two books, read them (you can thank me later), and then come back.
One of the key concepts in Team Topologies is the flow of change. Let’s call it flow. In a software production organisation that practices continuous delivery, flow is the constant stream of atomic changes that your stream-aligned teams produce. These small changes are constantly being rolled out to production, allowing stream aligned teams to get immediate feedback from the users. The feedback is useful to understand what users want and to eliminate bugs & security issues. Optimising your flow maximises the value your systems bring to your users & customers.
The problem
Security vs. control
Security professionals unfortunately conflate security and control.
Security, noun
protection of a person, building, organization, or country against threats such as crime or attacks by foreign countries
Control, verb
to order, limit, or rule something, or someone’s actions or behaviour
The thinking is straightforward: how can I guarantee my system is secure if I can’t control what goes in it? I can’t, so I’ll only allow into production what I have approved!
Security vs. compliance
Thankfully, we’ve got security standards, written by very smart people, to explain to mere mortals (aka non security people), how things should be! Am I right ladies and gentlemen?
One of SOC 2’s common criteria is Change Management. SOC 2 auditors usually check that:
- change requests based upon identified needs are evaluated, documented, and authorized by the business owner and IT management
- change is reviewed, tested, and approved for intended functionality and specifications by a peer programmer or an independent IT quality assurance team
- upon testing approval, the change advisory board (CAB) approves and schedules the change for deployment in the production environment.
Of course, the controls themselves are not set by the AICPA, so your mileage may vary.
On the other hand, the ISO/IEC 27001 standard’s Annex A includes:
8.32. Changes to information processing facilities and information systems shall be subject to change management procedures.
That’s too vague, so ISO/IEC 27002 follows up by adding that:
The change control procedures should include:
a) planning and assessing the potential impact of changes considering all dependencies;
b) authorization of changes;
I used to work for an organisation that had a CAB for each and every production change. Every change had to be approved by a development manager, an operations manager, and a security manager.
As luck would have it, each team was in a separate building. While the buildings were all located close to each other, the teams rarely mingled. Given the nature of their jobs, operations and security people liked stability because change is potentially risky.
For the change to go through, the CAB required consensus; it was the embodiment of design by committee, where the committee members had fundamentally diverging goals. That is not a good thing.
So, how do you securely ship value-adding products to your users in this situation?
You don’t.
Doing better
You don’t have to control every inch of a system for it to be secure.
The water analogy
Do you think water treatment engineers inspect every drop of water before it goes through the city pipes to your home? Do they have a huge dam that holds all the river’s water, and then they hold a Water Drop Admission Board to approve every drop of water that comes through? Of course not! That’s absurd!
Just in case, allow me to elaborate why that is absurd:
- There will never be enough engineers to inspect every drop of water;
- Running tests on all these water drops would be too costly;
- People need water to survive, and cutting the supply to carry out a long validation process is not an option.
Water safety engineers ensure that our drinking water is safe thanks to a combination of techniques: sedimentation, filtration, disinfection etc. (learn more here). All this happens before the water reaches consumer households. On top of all that, engineers frequently sample the water within the distribution systems, and run a plethora of tests to ensure all metrics are within the target ranges.
They’ve been successfully doing this for more than a hundred years. In my opinion, the flow of changes, from developer to user, in a software production organisation, is similar to the flow of water from source to consumer.
Let’s take a leaf out of the water safety management book, and apply a similar approach to software engineering, to improve the security of the software we ship to our users.
High performance secure software flow
You can achieve similar results to what water safety professionals have been getting for the last few decades by investing in a solid, secure platform.
Don’t fall into the Ivory tower trap. Do the engineering work to add security requirements into your organisation’s flow. The diagram above is a simplified version of what could be done, but you can do so much more!
Harness the power of platform engineering to create a secure golden path for your coworkers to follow. Develop automated security checks to reduce stream-aligned teams’ dependency on your security team; that frees both sets of teams to work on improving the overall flow.
Obviously, CI pipeline checks alone are not enough for optimum performance. That’s where the frequent “out-of-band” sampling comes in. Bug bounty & penetration testing are two examples of out-of-band checks you can do to continuously improve your overall security posture. However, they definitely should not be the core of your security program: they’re just too infrequent and slow. I’d argue it’s best to invest in SDLC-level security guardrails because they provide the fast feedback required for the best flow.
Learnings
If while, reading this, you got the idea that security compliance standards are a hindrance, please think again. They bring structure and security hygiene that are undeniably useful. The hindrance comes from security compliance professionals who are not able, or willing, to understand the complexities of the systems they’re auditing and how they’re evolving.
As a security professional, you should understand that a highly performant and well managed flow is entirely aligned with your objective of eliminating security bugs, and minimising risk. And I recommend that you show your auditor this “new” way of building systems.
This is the way.