Mar 25, 2022, Infrastructure, Web

Establishing an Enterprise DevSecOps culture

John Oxley Business Consultant
enterprise devops culture
A long time ago, I worked in the information security team of a large financial organization. We were completely isolated from the rest of the administration, literally stuck in an office at the end of a building wing, visitors were treated with suspicion and, in turn, we were treated with subtle contempt. After all, we were the protectors of the business, not the employees.

Access control, email intercepts, documenting security breaches, all those lovely things that seemed to hamper regular business operations. Back then, all departments were individual, with development, operations and security in their own silo mentality.

But this is today, and security has taken on a whole new dimension, with new vulnerabilities and intrusions popping up almost daily, and this has created a huge problem. As more and more businesses embrace cloud technologies, the need for security in development of applications has grown exponentially.

This is where the trinity of development, security and operations comes into play. They have to work together, but how to implement a shift in culture?

What is DevSecOps?

Development work is to plan, code, build and test new software to the point of releasing, then the Operational side takes over the release and deploys, operates and monitors the software. Surrounding this continuous flow is security, once placed at the end of the process but now moved to encompass complete oversight on all aspects, from planning to monitoring. 

DevSecOps allows security testing to be done concurrently with the design and implementation phases of development. Initially, security was placed at the end of the life cycle so as not to impede workflow, but it has been made clear that security needs to be an active partner, front and centre.

Why is it worthwhile?

In 2020, the Consortium for Information and Software quality estimated that poor quality software cost US companies over $2 trillion, mostly from operational software failures. And it is estimated that fixing software problems cost an additional 10 times more after release, compared to operating a robust security system in the planning-to-deployment phases.

As modern applications are perhaps more assembled than written, there is a big chance that the building blocks, downloaded from open source libraries, with custom code added by the developers, already contain flaws or vulnerabilities. According to Gartner, this could be 70% of open source software. All components, or packages, used to build the software must be scanned with application security tools before the project starts. The developers can then check to see if the software sticks to the specifications established by the business and mitigate risk.

Higher quality software can be achieved by running continuous reliability and performance tests with the software being developed, reducing costs by being able to spot defects earlier. Once the software is ready for release by the Ops team, security continues by testing a complete product for resilience, reliability and performance, and will fix any issues that might have been difficult to see in the development stages, such as real-world user load.

Shaping and implementing a DevSecOps culture

This is probably the hardest task. People. Change. A feeling of loss of control. 

Traditionally, most developers looked at security as an obstacle to avoid. They would rather have done their job and offloaded the completed software as smoothly as possible. And it was understandable, having a sense of personal pride in your work and not having to rely on others. But it is no longer viable. The security of the product must be the number one target.  

With this in mind, how does this new culture get implemented?

This is a big change in culture

Everyone needs to ‘buy in’ to make it work, and there must be plenty of empathy and understanding from the top down. Openness and ongoing learning, and security education and training with the right tools, will help in the transformation.

Everyone is equal

Except some are more equal than others. DevOps should not be steered towards the realm of security, rather the other way round. Security should be blended into DevOps so the workflow is not affected, and the balance within DevOps remains.

Security is not a lumbering beast

Demonstrate to the DevOps team that security will not hold them up, and will match the speed of the production environment. Create strong feedback loops to find opportunities to form a smooth pipeline.

A mixed bag

Create cross-functional teams, with different skill sets, allowing for faster adjustment and optimization of the development process.

Outcome over Output

A blinkered view of output will eventually turn development into an assembly line, taking a toll on morale and motivation. Outcome needs to show positive benefits for both the customer and the company, delivering a well crafted product with a sense of ownership.

Prioritise automation

Applying AI to Ops (AIOps) gives the ability to identify issues in real time, and in certain conditions is able to remediate them. AIOps automates key tasks and suggests precise solutions to most common issues.

In a perfect DevSecOps world, everyone will be responsible for security, but in order for it to happen, there must be a clear feeling of ownership within the company. A recent (2021) poll of Chief Information Security Officers revealed that over 70% of them were not fully confident that their code was free of vulnerabilities. Traditional security procedures just can’t keep up with the rapid pace of changing technologies.