MITRE ATT&CK Evaluation Reveals CrowdStrike Falcon® as the Most Effective EDR Solution

CrowdStrike delivers the most proactive detections, visibility, and context in MITRE ATT&CK™ Evaluation testing. At CrowdStrike, we believe strongly in third-party product testing and evaluations. Since 2016, we have proven the anti-malware capabilities of the Falcon Endpoint Protection platform by consistently participating in non-paid public testing with testing organizations such as AV-Comparatives and SE Labs. MITRE’s new ATT&CK-based product evaluation program is a critical moment for our industry because it introduces the first effective way to go beyond anti-malware testing and validate the EDR capabilities of endpoint protection platforms by simulating a real-world attack. On November 29, 2018, MITRE released the latest round of results from their MITRE ATT&CK™ Evaluations program. The CrowdStrike^® Falcon^® platform was validated for its ability to detect attack techniques employed by GOTHIC PANDA (also known as APT3), a sophisticated adversary affiliated with the Chinese government. During this evaluation, CrowdStrike’s solution delivered: the most proactive detections of any tested solution; the most complete out-of-the-box visibility; and the deepest context for each phase of the simulated attack. MITRE’s evaluation represents the industry’s most ambitious effort to date in assessing how security solutions perform in advanced, real-world attack scenarios, and CrowdStrike is proud to be a part of it.

CrowdStrike Falcon® Delivers

CrowdStrike’s results speak for themselves. MITRE’s evaluation validated CrowdStrike’s unprecedented ability to provide broad protection against sophisticated attacks. The results showed that CrowdStrike Falcon® excels at identifying hands-on-keyboard adversary activity across the entire ATT&CK framework, from the Initial Access stage, all the way to Exfiltration and Command and Control (C2).

Key Test Result 1: Proactive Detections

CrowdStrike delivered the most proactive detections of any solution in this round of validation¹, with nearly twice as many detections as the nearest competitor. This proves that while many endpoint detection and response (EDR) solutions are effective at collecting telemetry, they do a poor job of highlighting what is actually relevant for the end users. The challenge is to surface the right data, at the right time, during the heat of battle. CrowdStrike’s combination of machine learning, behavioral detection, and Falcon OverWatch™ managed hunting (MDR) is uniquely effective at shining a bright light on malicious activity for CrowdStrike’s customers. Examples of this included:

• CrowdStrike’s machine learning uniquely alerted on an initial malicious payload and provided proactive detection at execution time (attack stage 1.A.1).
• CrowdStrike’s behavioral analytics allowed us to be the only test participant to provide proactive detection on DNS-based C2 activity (attack stage 1.C.1).
• The CrowdStrike OverWatch team uniquely identified and alerted on deletion of file-based attack artifacts (attack stages 19.D.1, 19.D.2) and suspicious discovery activities (4.A.1, 4.A.2, 4.B.1).

Key Test Result 2: Out-of-the-Box Visibility

CrowdStrike provided the most complete out-of-the-box visibility². Falcon provided the broadest visibility into every stage of the simulated attack without requiring any product updates or “configuration changes” during the validation. CrowdStrike’s unique cloud-delivered EDR solution collects comprehensive endpoint telemetry continuously, not only when suspicious activities are happening. CrowdStrike takes great pride in not only delivering the best levels of detection, but also in our ease-of-use and ability to be working immediately without complex intervention by the user, which is critical in dealing with real-world attacks. This validation testing showed once again that nobody does it better. The following are some examples:

• Nearly all vendors in the evaluation were unable to provide basic visibility around creation of a new Windows user account, which CrowdStrike delivered easily (attack stage 7.A.1).
• Three of the participants were able to do so only after manually tuning configuration, and two others were completely unable to meet this basic visibility requirement.
• Many vendors continue to have visibility gaps around PowerShell-based threats — CrowdStrike offers unique capabilities here, including visibility into encoded PowerShell and exposed keylogger activity, which were missed by all other vendors (attack stage 15.A.1, 15.B.1).

*A key additional point on total visibility is that we are aware that the evaluation results show that we had visibility into 105 of the test cases and one other vendor had 108. However, some clarity is needed on how product changes (“configuration updates”) were implemented in this test. CrowdStrike took the position that modifying product capabilities during the test could call the results into question. To avoid this, we did not issue product updates during the test. Thus, out-of-the-box visibility for us was 105 and another vendor was 103. Since the test, we’ve been working to add additional visibility for the gaps identified. At the time of this writing, we have visibility into 109 with even more coming soon — they can be seen in a product demo or free trial.

Key Test Result 3: Attack Context

CrowdStrike provided the deepest context for each phase of the simulated attack. Many solutions put the onus of “connecting the dots” of an attack on the end user, which requires highly skilled analysts and increases response time dramatically. CrowdStrike’s approach is radically more effective. CrowdStrike was the only tested vendor to augment detections with timely insights from expert threat hunters. Falcon OverWatch emails provided environment-specific context, investigative support and remediation guidance so that the end user could quickly understand the tactics, techniques and procedures (TTPs) that were being used against them as the attacks unfolded. Only CrowdStrike’s unique combination of technology and expertise was able to deliver the highest degree of actionable information, making it exceptionally simple for security analysts to quickly understand the full context of the attacks. This comprehensive awareness and visibility is critical in allowing organizations to react to threats within the “breakout” time window.

The Importance of Real-world Testing

Independent, transparent and ethical third-party testing is critical in the cybersecurity world, as this kind of information is difficult, if not impossible, for end-user organizations to produce on their own. Testing products for security effectiveness means you need the resources and skills to push them to their limits in the same way that an adversary would in the real world. MITRE, a federally funded research and development center, is in a unique position to deliver open and ethical assessments of cybersecurity products under realistic conditions. The entire cybersecurity industry should applaud them for the contributions they’ve made in creating the MITRE ATT&CK framework, as well as for conducting these unique and unbiased evaluations. The crucial need for real-world testing is most dramatically illustrated in the U.S. auto industry, where independent, third-party crash testing is performed on every new vehicle to validate the safety claims of manufacturers. For years, the gold standard for this testing was the head-on collision — considered the worst-case scenario. These tests resulted in vehicle safety innovations that have saved countless lives, but not as many as expected. Further study showed that direct head-on collisions rarely occur, instead, vehicles in the real world tend to hit each other at an angle. In these cases, the safety measures, which were designed to perform very well under specific test criteria, did not perform as well in real-life situations. These findings have driven massive overhauls in how vehicles are tested, resulting in much safer vehicles on our roads and highways. Cybersecurity is going through a similar renaissance in testing, which CrowdStrike supports as a key effort needed to drive our industry forward. To date, testing organizations such as AV-Comparatives, AV Test and SE Labs have done excellent work in testing anti-malware products against the latest threats. These tests have, in turn, helped to drive and encourage innovations and improvements across the anti-malware industry. Today, leading vendors (including CrowdStrike) have proven track records of detecting 98 to 99 percent or more of tested threats. Despite these excellent test results, our news feeds are filled with stories of breaches on a daily basis. Why? Because malware represents only a part of the problem. Today’s adversaries are motivated and highly skilled, and they come armed with a wide range of tools — not just malware. The 2018 CrowdStrike Global Threat Report showed that 39 percent of attacks used no malware at all, proving that even an anti-malware solution that is 100 percent effective against malware threats still leaves massive gaps in protection. MITRE ATT&CK Evaluation is helping to fill this testing gap. MITRE’s evaluations take a full spectrum of TTPs observed from encounters with real-world adversaries (in this case, GOTHIC PANDA), and emulate these techniques in a repeatable manner. For the first time, MITRE’s evaluations make it possible to draw conclusions about how an endpoint protection solution would behave — not only against a piece of malware, but against the full range of tradecraft that is likely to be encountered in a targeted attack. Naturally, this is just the beginning. Our adversaries don’t stand still and neither can our industry. CrowdStrike is proud to be at the forefront of these efforts to validate the effectiveness of the security controls our industry relies on every day, in real-world conditions. We remain dedicated to the continuing fight to adapt and protect against what comes tomorrow.

Footnotes

1. A “proactive detection” is a test case identified by MITRE with a Detection Type of either “Generic Behavior” or “Specific Behavior”. MITRE credited CrowdStrike with 69 proactive detections out of 136 test cases.
2. “Out-of-the-box visibility” is defined as test cases identified by MITRE with a Detection Type of something other than “None” or “Configuration Change”. MITRE credited CrowdStrike with out-of-the-box visibility for 105 out of 136 test cases.

Learn More:

• Read test results on the MITRE ATT&CK Emulation website.
• Read the press release.
• Download the white paper: “Faster Response with CrowdStrike and MITRE ATT&CK.”
• See what third-party security testers and reviewers are saying about Falcon.
• Review the results from MITRE’s previous GOTHIC PANDA emulation testing with CrowdStrike.
• Test CrowdStrike next-gen AV for yourself: Start your free trial of Falcon Prevent™ today.