![Helping Non-Security Stakeholders Understand ATT&CK in 10 Minutes or Less [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/video-ATTCK2-1)
![Qatar’s Commercial Bank Chooses CrowdStrike Falcon®: A Partnership Based on Trust [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/Edward-Gonam-Qatar-Blog2-1)
![Endpoint Protection and Threat Intelligence: The Way Forward [VIDEO]](https://assets.crowdstrike.com/is/image/crowdstrikeinc/GK-Blog_Images-1)
MITRE’s recent ATT&CK® Evaluation test against Russian adversary COZY BEAR (aka APT29) has raised some interesting questions about the proper role of humans in detecting and understanding sophisticated threats. In the test, 15 out of 21 participants deployed some type of human-powered detection and response capability during the test, with wildly varying results. It raises the question: What should a good result look like?Use Case 1: Threat hunting
CrowdStrike threat hunters can uncover advanced tactics, techniques and procedures (TTPs) that aren’t surfaced clearly by the technology itself, and bring them to the attention of security analysts. Proactive human threat hunting takes the overall solution to the next level of efficacy, to see and stop sophisticated tradecraft that is designed specifically to evade automated detection techniques.Measuring the Value-add of Humans
MITRE’s evaluation measured how many test cases were detected or enriched by each vendor’s managed security service providers (MSSP), labeled by MITRE as “MSSP detections.” Given the statements above, what was the optimal number of MSSP detections in MITRE’s ATT&CK Evaluation? Answer: the minimum number needed to augment the technology, equipping security analysts with the information they need to see the threats clearly and make plans to take decisive action.
Service provider detections add enormous value when they help to surface information that is beyond the capabilities of automated technology. On the other hand, if you’re using humans to highlight TTPs that most products handle well with technology alone, or to deliver only basic insights and context, then your solution is behind the curve and you are hiding product failings with humans.
Figure 1: Falcon OverWatch reports on suspicious file collection activity within MITRE’s evaluation. (Click image to enlarge)
As the VP of CrowdStrike® Falcon OverWatchTM, I think this is a great opportunity to shed some light on how managed services participates in this type of testing, and explore what you can and can’t deduce from the results. It turns out that a lot of what you see from the humans depends on the capabilities of the technology that’s supporting them. If your technology is pulling its weight, throwing more human-powered alerts and reports over the wall to your analysts is not a better outcome — it’s just more work.
Humans Teaming With Technology
CrowdStrike pioneered the concept of managed threat hunting, and we have been providing that service through Falcon OverWatch since 2014. Falcon OverWatch sees and stops sophisticated threats by relying on elite human expertise and cloud-scale data to hunt relentlessly for advanced attacks that can otherwise go undetected. In the last year, the OverWatch team’s 24/7/365 operations have observed and disrupted more than 35,000 potential breaches.When OverWatch or other service providers are engaged in a testing situation such as the MITRE ATT&CK® Evaluation, there are two different ways we can show value.
Use Case 1: Threat hunting
CrowdStrike threat hunters can uncover advanced tactics, techniques and procedures (TTPs) that aren’t surfaced clearly by the technology itself, and bring them to the attention of security analysts. Proactive human threat hunting takes the overall solution to the next level of efficacy, to see and stop sophisticated tradecraft that is designed specifically to evade automated detection techniques.
Use Case 2: Enrichment
Humans can add information and context to existing endpoint detection and response (EDR) product detections and telemetry that aren’t surfaced cleanly by the technology itself. This can help improve prioritization, enabling analysts to properly deal with most important alerts first: alerts that otherwise would not have hit the top of the priority list. This human-powered enrichment also drives faster investigations and response. Enrichment saves analysts the time and burden of digging up information on their own, increasing productivity and allowing your organization to do more with the team you have.Measuring the Value-add of Humans
MITRE’s evaluation measured how many test cases were detected or enriched by each vendor’s managed security service providers (MSSP), labeled by MITRE as “MSSP detections.” Given the statements above, what was the optimal number of MSSP detections in MITRE’s ATT&CK Evaluation? Answer: the minimum number needed to augment the technology, equipping security analysts with the information they need to see the threats clearly and make plans to take decisive action.
Service provider detections add enormous value when they help to surface information that is beyond the capabilities of automated technology. On the other hand, if you’re using humans to highlight TTPs that most products handle well with technology alone, or to deliver only basic insights and context, then your solution is behind the curve and you are hiding product failings with humans.
Putting Technology and Humans to the Test
In this year’s MITRE evaluation, lots of vendors generated unprecedented numbers of MSSP detections, including some vendors who are very new to the threat hunting game. CrowdStrike Falcon® OverWatch, one of the industry’s most experienced threat hunting organizations, generated only a few results in MITRE’s testing. Why? Because it wasn’t necessary. MITRE’s simulated attacks emulated a known adversary, with well-known TTPs. There wasn’t anything here that was new or especially novel. EDR vendors have had years to understand how COZY BEAR has operated and ensure their products have proper detection across the full range of known tradecraft. The CrowdStrike Falcon®® platform did its job as expected, so threat hunting wasn’t necessary to fully surface most of the relevant TTPs. CrowdStrike’s new CrowdScore™ capability was another game-changer for this test, using the power of the CrowdStrike Threat Graph® and sophisticated analytics to pull together context and “connect the dots,” building a more complete picture for defenders. Again, CrowdStrike technology did its job in an efficient and effective manner, so human intervention wasn’t necessary to connect telemetry to alerts. Every step of the way, Falcon OverWatch maintained its standard high level of vigilance, staying on continuous lookout for hidden threats. Throughout it all, OverWatch successfully avoided bombarding analysts with unnecessary and repetitive alerts and reports.This approach delivered the best possible outcome for a security team. Unfortunately, when other solution providers adopt the philosophy that “more is always better,” it becomes a real challenge to meaningfully compare MSSP results in MITRE’s testing. This is why Forrester Principal Analyst Josh Zelonis said in a recent blog, “The evaluation contains a mini game of whack-a-mole, but you should ignore all the MSSP stuff.”
Figure 1: Falcon OverWatch reports on suspicious file collection activity within MITRE’s evaluation. (Click image to enlarge)Let Humans Do What Humans Do Best
There remains a lot of ground where human experience, creativity and intuition provide critical contributions to defenders, which are challenging to surface in this type of test. If your organization is considering working with a managed services organization to help with detecting and responding to threats, here are some questions you might ask:- Can the managed service identify new and novel TTPs? Falcon OverWatch employs an elite staff with a variety of backgrounds including defense, intel and commercial enterprise. We arm them with unparalleled global telemetry and threat intelligence. The OverWatch team leverages these tools to deliver laser-focused, proactive threat hunting, to help customers see and stop hidden, sophisticated threats every minute of every day.
- How well does the managed service perform in a real-world environment that’s full of benign traffic? MITRE evaluation testing is performed as an open book. All activity in MITRE’s evaluation was malicious, and it is relatively simple to find and report on the bad stuff. The real value in a managed threat hunting service is surfacing stealthy threats in an ocean of benign activity. For more than six years, Falcon Overwatch has provided relentless protection for CrowdStrike customers in real-world situations.
- Does the managed service operate 24/7, around the globe? Falcon OverWatch hunts 24/7/365, because your adversary does not sleep and does not take holidays.
- How long does it take for the humans to detect an emerging threat and respond with information your team can digest and act on? Can you expect a report in hours? Days? Falcon OverWatch provides continuous vigilance and delivers notifications in near real-time, enabling your team to respond quickly to stealthy intrusions before they become a breach.
- How and when are results delivered to your analysts? A 50-page PDF describing an attack the day after it happened provides zero value in stopping an in-progress intrusion. Falcon OverWatch delivers real-time alerts directly in the Falcon console, as well as email alerts for urgent hands-on-keyboard activity, ensuring that your staff has the information they need, in a timely fashion and actionable format.
Additional Resources
- Read the SANS review of Falcon OverWatch to understand how OverWatch delivers consistent results with speed and efficacy.
- Read the MDR Buyer’s Guide for information about how managed services can help your organization detect and respond to all types of threats.
- See how CrowdStrike performed in the recent MITRE ATT&CK Evaluation.
- Hear from actual customers about how the Falcon platform with OverWatch managed threat hunting has changed their ability to stop breaches.
