Shift Left, Measure Right: Assessing the Efficacy of Application Security in the Age of CI/CD

November 21, 2024

| Jamie Gale | Cloud & Application Security

As organizations continue to invest in developing cloud-native applications, they must also invest in securing them. Industry data shows application security is a growing market with investments predicted to reach $13.57 billion USD by 2029, according to Statista Market Insights.

Figure 1. Investments in application security predicted to climb to $13.57 billion USD by 2029 (Source: Statista Market Insights)

But application security is a tricky process to navigate for many DevOps and DevSecOps teams. Today’s applications are constantly evolving with new features and updates, continuously introducing the possibility of vulnerabilities and misconfigurations that could heighten risk. Further, organizations navigating the transition from DevOps to DevSecOps may lack the metrics needed to effectively track and measure their application security posture.

Here we discuss the challenges in securing cloud-native applications and identify ways to measure the efficacy of an application security strategy in reducing risk.

Security Starts Left but Doesn’t End There

Integrating security early and frequently in the software development lifecycle and CI/CD pipeline — a process known as "shifting security left" — is an application security best practice. However, modern applications constantly evolve and frequently receive updates after they’re deployed. According to the CrowdStrike 2024 State of Application Security Report, 71% of organizations update applications weekly, and 19% do so multiple times a day.

Application security teams must continually review application code changes as they are developed and deployed to confirm they are free of known vulnerabilities, misconfigurations and other security problems. To accomplish this, application security teams use a variety of scanning and testing tools to uncover potential issues in the application code, configuration or runtime execution.

While these tools are essential for secure development, there must be a balance across the entire lifecycle of the application. Pre-production security tools can either miss issues altogether or generate an abundance of false positives. For example, software composition analysis (SCA) tools find vulnerabilities in open source libraries but are unable to identify vulnerabilities in custom code. Static application security testing (SAST) is valuable for analyzing custom code but cannot detect runtime configuration issues and business logic errors. Conversely, dynamic application security testing focuses on discovering vulnerabilities that occur during execution, such as injection attacks or misconfigurations, but it does not have visibility into the source code, meaning it may miss vulnerabilities like hardcoded credentials, logic errors or insecure coding practices that are only detectable by SAST.

Measuring the success of an application security strategy shouldn't focus solely on the number of detected vulnerabilities or security issues, or on remediation speed. Rather, success should be measured by managing risk and prioritizing the most impactful issues.

Moving Beyond DevOps KPIs

DevOps, initially focused on accelerating software development and delivery, has for many organizations evolved into DevSecOps, which is responsible for integrating security into the CI/CD process.

As DevOps evolves into DevSecOps, the metrics used to track and measure success must also change. Google’s DevOps research assessment (DORA) team tracks the following five metrics to determine DevOps success and maturity: deployment frequency, lead time for code changes, mean time to recovery, change failure rate, and reliability. While many industry leaders use DORA metrics to measure DevOps success, these tend to focus on the volume of vulnerabilities and the speed of resolution rather than the efficiency and effectiveness of resolving security risk. They lack the specific application security metrics necessary for DevSecOps.

There are several additional metrics DevSecOps teams must consider when measuring the security of their applications.

Making Sense of Application Security and CI/CD

To measure application security effectiveness, teams must identify the processes and tools embedded in their CI/CD pipeline, then evaluate the coverage and gaps spanning stages from pre-production to production.

Figure 2. Application security across CI/CD

The next step is to find the KPIs to measure the effectiveness of your application security across the CI/CD pipeline. Below we highlight some KPIs that can help measure the impact that application security tools and processes have on risk.

In the following examples, a critical risk is any threat that is exploitable and has access to sensitive data.

Table 1. Application security KPIs (pre-production)
Pre-Production
KPI	Description	Purpose
Critical Risk Volume	# of critical risks detected in pre-production	To understand the volume of critical risks found in pre-production (required for the Critical Risk Escape Rate KPI)
Critical Risk Remediation Queue	# of critical risk tickets	To understand the stress levels and workload placed on the developers who need to fix these issues
Known Risk Acceptance	# of known risks not remediated (accepted) and pushed to production with reason	To understand how many risks are accepted

Table 2. Application security KPIs (production)
Production
KPI	Description	Purpose
Critical Risk Volume	# of critical risks detected in deployed apps	To compare to the volume of critical risks found in pre-production app security (required for the Critical Risk Escape Rate KPI)
Critical Risk Escape Rate	Ratio: # of critical risks found in pre-production to # found in production	To assess effectiveness of pre-production security
Critical Risks Per Deployment	Average # of critical risks for each deployment	To benchmark expected critical risks
Mean Time Between Critical Risks	Average time between critical risk detection	To assess critical risk prevention/reduction

Tracking these metrics provides valuable insights into the effectiveness of application security.

Assess Qualitative Metrics Focused on Teams

DevSecOps is both a technical and cultural endeavor, so it’s important to measure the sentiment of key stakeholders. Here are some questions to consider.

Developer Experience

Do you have the information you need to fix security bugs in code efficiently?

Security Experience

How much time do you spend on manual security reviews and checklists?
Do you have time to take on proactive security projects?

SRE/Operations Experience

Do you have the necessary tools and processes to quickly pinpoint the root cause of an outage and restore service?

CISO Experience

How secure are your applications today?
In the event of a breach related to an application, do your teams have the information and tools they need to respond quickly?

Closing the Gap with Falcon ASPM

The majority of today’s application security tools and processes take place before an application is deployed. Early inclusion of security, or shifting security left, will always be a best practice, but it’s not enough. Application security is essential for every phase of the development life cycle, including — and most importantly — production.

CrowdStrike Falcon® Cloud Security is helping organizations extend their cloud security to applications running in production with Falcon Application Security Posture Management (ASPM). Falcon ASPM is a risk-driven, agentless approach for securing complex, frequently changing applications built and run in the cloud. Falcon ASPM assesses risk in a unique way that helps teams identify the greatest threats in their applications. Here are some of the most important factors that drive Falcon ASPM risk scoring:

Severity of the threat(s) in a given application microservice
Connections to sensitive or proprietary data
Exposure to the internet, third parties or other connections
Reachability (for code library vulnerabilities) of threats to the application

With better risk scoring comes better prioritization. Instead of another application security tool that leaves you with hundreds of potentially critical issues to fix, Falcon ASPM automatically narrows down the top few real risks for you and gives your developers the information they need to fix those issues quickly.

Ready to see how you measure up? Get a free cloud security health check.