Inside the MITRE ATT&CK Evaluation: How CrowdStrike’s Elite Managed Services Operate in the Real World

Following CrowdStrike’s strong performance in the first-ever MITRE ATT&CK^® Evaluations for Security Managed Services Providers with 99% detection coverage, we take a deep dive into the testing process and how our elite managed services operate in the real world.

We recently announced CrowdStrike achieved 99% detection coverage in the inaugural MITRE ATT&CK Evaluations for Security Managed Services Providers. These results speak to our industry-leading technology and elite team of human experts, which combined detected 75 of 76 adversary techniques tested.

Closed-book testing, in which no vendor has advance notice of the adversary, methodology or specific timing of the test, provides the closest reflection of how a security vendor can protect a customer against a real-world threat. The CrowdStrike Falcon^® Complete and CrowdStrike^® Falcon OverWatch™ managed threat hunting teams, leveraging the power of the CrowdStrike Falcon^® platform, identified the adversary and associated tradecraft within minutes.

Human-led managed detection and response (MDR) and threat hunting deliver a level of protection autonomous tools alone cannot. While many MDR solutions tested in the MITRE ATT&CK evaluation are entirely automated, Falcon Complete rises above the MDR competition with its powerful combination of security technology and human expertise. Our MDR analysts, threat hunters and incident responders have decades of experience in triaging, investigating, and remediating threats. This, combined with CrowdStrike’s integrated threat intelligence and tooling, enables them to find and stop malicious activity at unrivaled speed and scale.

Falcon OverWatch — our proactive, 24/7 threat hunting service — is a core component of Falcon Complete and played an integral role in the MITRE ATT&CK Evaluations for Security Managed Services Providers. Falcon OverWatch threat hunters were involved throughout the evaluation and were essential to CrowdStrike identifying and reporting 75 of the 76 tested steps.

Watch this short video to see how Falcon OverWatch proactively hunts for threats.

Inside MITRE ATT&CK Testing with Falcon Complete

We approached this MITRE evaluation with the same configuration and processes we use with every Falcon Complete customer. Onboarding begins with an orientation call during which a specialist reviews the typical Falcon Complete operating model, establishes roles and responsibilities, and determines acceptable tolerance levels to carry out various response and remediation actions on behalf of the customer. For the MITRE ATT&CK evaluation, onboarding took place before the start of the active testing period.

Falcon Complete works closely with the customer to deploy the single lightweight Falcon agent onto thousands of systems, servers and machines across the enterprise. Once the agent is fully deployed, CrowdStrike takes over, handling all ensuing system configuration and optimization, and providing 24/7 threat monitoring, investigation and response. In a sense, Falcon Complete acts as an embedded extension of the customer’s security team, continuously maximizing its investment through more effective and efficient security operations and outcomes.

When the active five-day testing period began, CrowdStrike was left in the dark (as was every vendor participant), unsure when MITRE Engenuity’s attack emulation would start. The first malicious event of the 2022 MITRE ATT&CK Evaluations for Security Managed Services Providers began as many attacks do: with initial access attempts. The emulated adversary first gained entry by deploying a spear-phishing email that contained a malicious link and well-crafted social engineering pretext based on the target user’s job title to successfully dupe the intended user into clicking and launching the payload. Within minutes, CrowdStrike identified and triaged the first malicious activity. We quickly decoded the attacker’s XOR cipher to intercept and read the C2 communications, and used intelligence to identify the emulated nation-state adversary — known as OilRig by MITRE, or HELIX KITTEN as tracked by CrowdStrike Intelligence.

To keep customers informed of updates like these during an investigation, Falcon Complete uses a built-in module called “Message Center” within the Falcon console to communicate in near-real time.

Figure 1. A screenshot of the Message Center inbox where customers receive updates from Falcon Complete (click to enlarge)

The team usually sends a brief message after its initial triage to share an early assessment of the incident, provide situational awareness and create a channel where they can continue collaboration with the customer. Over the course of an investigation, Falcon Complete will follow up to provide updates on findings, analytical details, recommended recovery actions, remediation activity and more.

Falcon Complete typically takes remediation action on behalf of our customers. The team does not merely notify and advise, but goes several steps further to to contain adversaries and clean systems of residual artifacts related to an intrusion — all within a series of pre-approved countermeasures. These remediation actions include, but are not limited to: removing files and persistence mechanisms, network containing systems, forcing MFA or password resets (for CrowdStrike Falcon^® Identity Threat Protection (ITP) customers). The rules of this year’s MITRE evaluation prohibited any remediation action, so in this case, the Falcon Complete team performed a purely descriptive role.

Figure 2. A message from the Falcon Complete team with communication regarding recovery and remediation actions (click to enlarge)

Detecting Adversary Techniques with Falcon OverWatch

Throughout the MITRE evaluation, CrowdStrike’s technology and human expertise were quick to detect the emulated adversary’s steps to mimic a legitimate attacker. On day three, for example, the adversary paused much of its activity using various persistence techniques in an attempt to stay hidden and lure defenders into a false sense of security.

The simulated adversary also executed a variety of reconnaissance commands in the Discovery phase of their attack chain, similar to how a real-life adversary would operate. Detecting this activity is critical because it provides an early indication an intruder has compromised a system and is attempting to get “the lay of the land,” during which threat actors may learn potential targets for privilege escalation and lateral movement.

CrowdStrike’s ability to detect these discovery steps is crucial. Falcon Complete identified all of these steps during the emulated adversary’s Discovery phase by using insight provided by the Falcon OverWatch team, as well as the Falcon Sandbox and integrated CrowdStrike threat intelligence.

In the MITRE testing environment and in real-world intrusion scenarios, the Falcon OverWatch team works hand-in-hand with Falcon Complete. When there is an indication of an adversary in an environment, Falcon OverWatch quickly identifies what is abnormal, notifies the Falcon Complete team and sends a detection trigger to the Falcon UI. Once Falcon Complete knows of the activity, the team scopes and contains the intrusion, reducing dwell time for adversaries. With the incident identified and threat actor eradicated, they provide the customer with recommendations and remediation steps to ensure the adversary does not return.

One factor not evaluated during this evaluation was CrowdStrike’s ability to detect threat activity in environments large and small. The MITRE testing environment only consisted of a few machines; however, our common enterprise environments have tens to hundreds of thousands of devices — and Falcon Complete can operate on all of them.

At the conclusion of the simulated intrusion, CrowdStrike provided a final report to MITRE, as the customer, describing the extent of the attack with meaningful context and analysis of all associated threat activity that took place. This report, which summarized the intrusion as part of the incident response, is typically a separate service provided by CrowdStrike Professional Services and was specifically developed for this evaluation. It included a consolidation of Message Center content along with our investigation’s key findings, a full investigative summary and timeline, recommendations and countermeasures, mappings to the MITRE ATT&CK framework, network diagrams, observed indicators of compromise (IOCs), tactics, techniques and procedures (TTPs), and more.

Exploring the Full Power of Falcon Complete

It’s important to note this MITRE ATT&CK evaluation did not test prevention or remediation capabilities, which are standard components of Falcon Complete. If CrowdStrike Falcon® Prevent next-generation antivirus and Falcon ITP were enabled for this test, we would have been able to demonstrate the full power of Falcon Complete by preventing initial access and stopping the attack before it started.

As important as it is to quickly detect threats, it is equally crucial to quickly begin remediation to stop adversaries in their tracks and prevent an intrusion from escalating. Another element missing from the MITRE ATT&CK evaluation was remediation.

Most MDR offerings stop after they identify and investigate a threat, leaving the customer to determine and execute a response. Some go one step further and disrupt attacks by performing containment countermeasures.

Falcon Complete takes this to the next level and performs surgical remediation of compromised endpoints. Falcon Complete works in close collaboration with Falcon OverWatch, takes their findings and immediately begins securing the victim environment. With Falcon Complete’s efficiency in remediation, it’s no exaggeration to say CrowdStrike would have run circles around the evaluation — widening the already substantial gap between our results and those of our competitors.

The 2022 MITRE ATT&CK Evaluations for Security Managed Services Providers demonstrates how CrowdStrike’s powerful combination of security technology and human experts lead the industry — and it doesn’t even cover CrowdStrike’s whole MDR story. What these MITRE ATT&CK results don’t show is how CrowdStrike’s MDR customers see even greater value from other included capabilities that were turned off; for example, Falcon Identity Threat Protection and end-to-end remediation of every detected event. In an upcoming CrowdCast on Dec. 8, our experts will discuss how security decision-makers can use the MITRE findings to inform their technology decisions and drive better security outcomes.

The industry-leading CrowdStrike Falcon platform sets the new standard in cybersecurity. Watch this demo to see the Falcon platform in action.

Additional Resources

• See the full results of the MITRE ATT&CK evaluation.
• Check out our Twitter Spaces discussions (Part 1 and Part 2) to learn how you can make the most of the new MITRE ATT&CK evaluation and hunting threats with Falcon OverWatch.
• Learn about another recent MITRE evaluation of the CrowdStrike Falcon platform in this blog: CrowdStrike Achieves 100% Prevention in Recent MITRE Engenuity ATT&CK Evaluation Emulating Russia-based Threat Groups.
• CrowdStrike has been named a Leader in the IDC MarketScape for MDR 2021 vendor assessment, highlighting the strengths of Falcon Complete, CrowdStrike’s strong cloud-native endpoint protection platform and continuous human threat hunting services.
• See what others are saying about CrowdStrike — visit the CrowdStrike Industry Recognition and Technology Validation webpage.
• Learn more about the powerful, cloud-native CrowdStrike Falcon^® platform.
• Get a full-featured free trial of CrowdStrike Falcon Prevent™ and see for yourself how true next-gen AV performs against today’s most sophisticated threats.