![Helping Non-Security Stakeholders Understand ATT&CK in 10 Minutes or Less [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/video-ATTCK2-1)
![Qatar’s Commercial Bank Chooses CrowdStrike Falcon®: A Partnership Based on Trust [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/Edward-Gonam-Qatar-Blog2-1)
![Endpoint Protection and Threat Intelligence: The Way Forward [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/GK-Blog_Images-1)
We couldn’t be prouder that CrowdStrike achieved the highest coverage across the last two consecutive MITRE Engenuity ATT&CK® Evaluations. We achieved 100% protection, 100% visibility and 100% analytic detection coverage in the Enterprise Round 5 evaluation — which equates to 100% prevention and stopping the breach. We also achieved the highest detection coverage in the Managed Security Services Providers testing.
Figure 1. CrowdStrike detects 143 (100%) steps during the MITRE Engenuity ATT&CK Evaluation: Enterprise Round 5 with high-quality analytics (Tactic and Technique). Updated November 2023.
Not all results are equal, which is hard to see in a comparative chart like this, as vendors have the opportunity to tune their systems in advance and apply configuration changes on-the-fly with teams of experts who may be working behind the scenes 24/7 during the testing period. For instance, we’ve seen vendors make updates to operating systems for the test, while others manually fix verdicts or add new context and detections.
Round 5 emulated Turla, which CrowdStrike classifies as VENOMOUS BEAR, a sophisticated Russia-based adversary. Given their advanced tactics, few vendors were able to identify all of their tradecraft, with the average visibility being 83%. High-quality analytic detection of Tactic and Technique were even less, with the average dropping to 66% — with CrowdStrike achieving full 100% coverage with analytic detections.
High-quality analytics are extremely important, as they provide insight into what an adversary is attempting to achieve and how they are attempting to achieve it. High-quality analytic detection provides the context that analysts need, letting them spend less time trying to determine if the alert is a true or false positive, and also provides insight into what an adversary is trying to do. With tactic and technique detections, security analysts can spend more doing what matters: stopping breaches.
Figure 2. CrowdStrike detected 99% of adversary techniques during MITRE ATT&CK Evaluations for Managed Security Services Providers.
To find the cybersecurity partner for you, it’s worth reviewing and correlating performance across many different tests that use different TTPs and force products to behave differently to find the true outcome of the platform. Ensure you look at the results of both open-book and closed-book tests, including those that measure false positives and performance, and know exactly what vendors did to achieve their results. Most importantly, make sure you can achieve those same outcomes in your enterprise. Sophisticated adversaries don’t provide the luxury of a heads-up, and customers won’t have potentially dozens of people working behind the scenes on their deployment in the real world.
However, interpreting the results of the Round 5 test can quickly become very confusing, with endless representations of test results from every provider. Unlike other third-party analysts, MITRE doesn’t place vendors on a quadrant or graph, or provide a comparative score. It leaves interpretation up to each vendor and customers themselves — meaning you’ll be flooded with claims of “winning” the evaluation.
In MITRE, there are no winners or leaders, only raw data on a vendor’s coverage against either a known or unknown adversary. Without better guidelines and enforcement from MITRE, the results will continue to confuse customers, given the wildly different solutions being tested and approaches to the evaluation. Evaluations like MITRE can help clarify your choice. We use the evaluations to further sharpen the capabilities of the CrowdStrike Falcon® platform, as well as ensure our customers understand our point of view on cybersecurity: Stopping the breach requires complete visibility, detection and protection that you can actually use in a real-world scenario.
How Should You Interpret the Results?
First, it’s important to understand the nuances of the two types of evaluations run by MITRE: open-book and closed-book tests. Open-book testing for known attackers: The MITRE ATT&CK Enterprise Evaluations, such as the recent Round 5, give vendors months of advance notice on the adversary being emulated and their tactics, techniques and procedures (TTPs), and then measure for coverage in a noiseless lab environment.
Figure 1. CrowdStrike detects 143 (100%) steps during the MITRE Engenuity ATT&CK Evaluation: Enterprise Round 5 with high-quality analytics (Tactic and Technique). Updated November 2023.
In a comparative chart like the one above, it isn’t possible to see if the capability provided is noisy annotated telemetry or important context added to a high-fidelity alert. Closed-book testing for unknown attackers: MITRE’s Managed Security Services Providers test is a truer measure of how vendors will protect a customer in the real world, with no do-overs or chances to hunt for additional evidence. The only notification vendors receive in advance is a start date, with no visibility into the adversary being emulated or their TTPs. MITRE runs the test, and you get a coverage score.
Figure 2. CrowdStrike detected 99% of adversary techniques during MITRE ATT&CK Evaluations for Managed Security Services Providers.Stopping Breaches Matters
Next, it’s critical to evaluate how effectively a vendor can stop adversaries without manual intervention. In the open-book Round 5 test, the average blocking rate was 86%, compared to CrowdStrike’s 100% protection. Even more important than the coverage is understanding how the scores were achieved. When digesting the MITRE results, ask vendors these three questions, and ask them to prove it:- Did they use easily bypassed signatures or custom detections requiring prior knowledge?
- Are the analytic detections and protections high-fidelity and suitable at enterprise scale?
- How can I reproduce this result in my own environment?
How Do You Bring It All Together?
At the end of the day, how a platform achieved its results matters as much as coverage itself. With open-book tests like the Enterprise Evaluation Round 5, you could hire enough experts to manually add custom tags, detections and context to achieve perfect coverage. That’s why you’ll see vendors shouting their coverage from the rooftops — as at face value many did well. All comparative charts, including the ones we’ve shown above, only tell part of the story. What’s important is looking at the details: how you do it matters as much as what you do. If you can’t actually achieve the results in your environment, it’s simply a number on a comparative chart. It can’t stop adversaries and it can’t stop breaches.Ask your provider, including us, how they achieved their scores — and ensure it wasn’t a herculean manual effort that could never work in the real world. It’s also important to understand exactly what the full bill-of-materials looks like to reproduce the results. Some vendors require a complex point product deployment, others an expensive combination of software and network security hardware, and others a significant headcount investment to operate.
The factor to consider most carefully are vendors that use custom test configurations that are impossible to reproduce in a real-world production environment. With CrowdStrike, our platform will always be delivered via our single lightweight agent that’s easy to deploy, easy to manage and never requires a reboot. We consolidate cybersecurity, with better outcomes, at a much better ROI. We stand behind our platform and the way we delivered our superior coverage across both MITRE’s open-book and closed-book testing for known and unknown adversaries — providing true breach prevention for the real world.
We encourage everyone in the industry to follow MITRE’s intention: Its testing yields valuable raw data that needs to be applied in your environment — with the context around how a vendor achieved its results — to be meaningful. And to our friends at MITRE, the time is now to shut down the endless noise and ensure customers understand your purpose: to make the world safer with better-informed decisions. If you want to learn more about using MITRE for your organization, register for our virtual event: MITRE Engenuity ATT&CK Evaluations: Inside the Enterprise Round 5 Results.
Additional Resources
- Learn more about the results of the MITRE Engenuity ATT&CK Evaluations: Enterprise in an upcoming CrowdCast.
- Read the MITRE Engenuity ATT&CK® Evaluations: Enterprise Round 5 press release and blog.
- Read the MITRE ATT&CK Evaluations for Security Services Providers press release.
- Check out our Unstoppable Innovations CrowdCast series to learn more about how you can rise above the adversaries and shut down breaches.
- CrowdStrike is ushering in the future of cybersecurity. Learn more about the lineup of announcements we made at Fal.Con 2023.
- Get a full-featured free trial of CrowdStrike Falcon Prevent™ and see for yourself how true next-gen AV performs against today’s most sophisticated threats.
