Confessions of a Responder: The Hardest Part of Incident Response Investigations

December 20, 2018

| | From The Front Lines
It’s not the disk forensics. It’s not the log analysis. It’s not even the lawyers (we love working with law firms!). It’s the lack of an effective technology deployment strategy. CrowdStrike® Services has observed organizations — both large and small — continue to encounter difficult and sometimes insurmountable challenges when trying to deploy endpoint security technology across their own environments. This blog will discuss the importance of deploying an endpoint technology to scope the incident, some common deployment challenges, and what organizations can do to overcome these challenges and be better prepared if and when an incident occurs.

Background

As part of incident response (IR) investigations, CrowdStrike Services provides the CrowdStrike Falcon®® platform, a real-time endpoint detection and prevention solution; the Falcon Forensic Collector, a historical endpoint artifact collection technology; and Falcon Network Sensors, a real-time network monitoring solution. This technology is deployed in the organization’s environment to rapidly scope the extent of current and historical adversary activity. In previous investigations, CrowdStrike Services has worked with a wide range of organizations from small and mid-sized companies to large enterprises with hundreds of thousands of systems. Regardless of size, the Services team has been able to quickly deploy the lightweight cloud-native Falcon platform to almost 100 percent of systems in a matter of a few days. For example, with one client, we were able to deploy the Falcon platform and Falcon Forensics Collector to over 70,000 systems in only four days. In these cases, this streamlined deployment was made possible because the organizations had the following network management components already in place:
  • An understanding and complete inventory of network assets in their environments
  • Infrastructure tooling to allow for the effective deployment of software to their endpoint infrastructures
  • A tried and tested overall game plan with backup plans for technology deployment
  • A single deployment team — or for larger clients, a fully integrated cross-functional team — responsible for deploying technology to the environment

Why It Matters

Deploying EDR (endpoint detection and response) technology, which is part of the CrowdStrike Falcon® platform, and Falcon Forensic Collector allows incident responders to quickly scope the extent of adversary activity. It also has the ability to detect new activity and react to the adversary with real-time prevention —

 

it can even provide network containment of the system, directly through the technology. On more than one occasion, we were engaged in a “firefight” with live eCrime or nation-state adversaries. During these active investigations, we were able to leverage the Falcon platform to create custom detection and prevention templates on-the-fly, providing rapid response and protection to our clients. In addition to real-time detection and prevention, the Falcon Forensic Collector allows us to scope historical activity on the systems to not only identify other systems the adversary had compromised or accessed, but also identify the earliest evidence of adversary activity in the environment, which leads to the identification of the initial infection vector, or how the incident began. Outlining the entire historical picture helps organizations understand how and why the adversary was successful, and what they can do to better defend themselves going forward. These are just some of the reasons why deploying these technologies is crucial to any investigation. Sometimes a sports analogy helps: In American football terms, let’s first take a look at some of the common challenges, or “penalties,” that can hurt organizations during the technology deployment phase of IR investigations.

Flags on the Play (Aka Common Challenges)

Some common challenges to deploying technology range from the simple to the rather perplexing. Murphy’s Law (which dictates that “whatever can go wrong, will go wrong”) applies to almost anything, and technology deployment is no exception — especially during an emergency situation such as incident response. Most of these challenges can be resolved through performing routine practice scenarios and exercises, and having a plan B or C to execute if your organization's initial deployment application or strategy fails. So let’s call a timeout, study some common “penalty plays,” and level the playing field with workarounds for each.

Ineligible Receiver Downfield (Aka Limited Asset Management)

Like a lineman on an American football team going downfield on a pass play, IT networks often have systems that either go places they shouldn’t, or show up on a network unexpectedly.

 

One of the most common challenges we encounter during an IR investigation is asset management —

 

specifically, when organizations do not have an understanding of what systems are in their environment. This may seem unlikely, but it is more common than you might think. There is usually a small, but sometimes large, number of systems that are either unknown or unmanaged in organizations. Unknown and unmanaged systems are especially common for organizations that have gone through mergers and acquisitions, or have large numbers of domains in their Active Directory Forest(s). These systems may later be discovered throughout the course of the investigation, or by tools such as Falcon Discover™, that will identify neighboring systems.

 

Once an unmanaged system is identified by Falcon Discover, the organization can follow up to make sure the appropriate security tools are installed on that system, to ensure its protected against an attack.
Figure 1: Sample overview of applications, assets and accounts with Falcon Discover (click for larger image)

 

Figure 2: Sample overview of managed and unmanaged assets with Falcon Discover (click for larger image)
Unmanaged systems typically have to be accessed through scripts, with assistance from the end-user or physical access by IT staff required to deploy software on them. Limiting both of these scenarios to a minimum should be a top goal for an organization’s IT department and will help the technology deployment process. Organizations can minimize the amount of unknown and unmanaged systems in their environments through commercial, open-source or even free utilities, such as Nmap, Spiceworks, and Advanced IP Scanner, to name a few. Keys to avoiding common pitfalls includes generating consistent and frequent asset scans, and having an asset management policy and plan in place to understand what systems are on the network. It’s also important to make sure they’re able to be managed or accessed, so technology can be deployed to them.

False Start (Aka Lack of Scalable Technology Deployment)

It takes a coordinated effort to snap the ball and begin a play in American football.

 

Everyone needs to know the count to “go.”

 

Similar to the snap count, an organization needs common tools to get started with technology deployment at scale. Too often, we encounter challenges because organizations do not have a commercial software management utility such as System Center Configuration Manager (SCCM), which can be leveraged for software deployment or patch management. SCCM, GPO, and other popular software managers like Jamf, Chef, or BigFix, are used with a high degree of success in IR scenarios where deploying a binary or script is needed. There are also free and open-source ways to accomplish rapid deployment of software for IR investigations. Leveraging bash scripting in Linux and MacOS, or PowerShell in Windows, are some options to achieve this. Below is an example of a PowerShell script to deploy WindowsSensor.exe for each line in servers.txt, where servers.txt contains a list of system names or IP addresses.
Figure 3: Sample PowerShell script to deploy WindowsSensor.exe to each hostname in servers.txt

 

If you’re “old school,” here’s an example of deploying via a batch script. In this example, WindowsSensor.exe is hosted on a local SMB share. We'll mount that share, execute the installer, and unmount the share via a batch file using the pushd and popd commands. First, we create a .bat file with the following contents: Then we execute .bat on systems with PSEXEC or other: In addition to native scripting, PDQ Deploy (https://www.pdq.com/pdq-deploy/) is software with a free license that organizations can leverage to create multi-step deployments (.exe, .msi, .bat, .ps1, .vbs) remotely and silently to the end user. PDQ Deploy allows remote execution of commands and scripts (PowerShell, VB, .bat) to deploy to computers and groups in PDQ Inventory, Active Directory, and Spiceworks. It even allows for deployment reports to keep track of which systems had a successful deployment. The free version of PDQ Deploy will only deploy to systems that are currently online and awake, without the ability to schedule deployment. The enterprise version has wake-on-LAN and scheduling, which will be a welcome addition for medium to large-sized organizations.

Too Many Men on the Field

 

(Aka Siloed Operations)

In American football, you’re allowed 11 players on the field at any time. A 12th man on the field causes confusion — he shouldn’t be there.

 

Similarly, organizations need to appropriately manage the number of people involved in technology deployment

 

and make sure all persons know their roles and have the ability to work together in a time-efficient and effective manner. Especially In large organizations, separation of duties is common, where multiple teams are used to deploy technology based on operating systems, sub-organizations or geographical locations. Unfortunately, this approach makes technology deployment much more complicated and error-prone. Having separate and sometimes siloed teams own deployment for certain systems increases the chances that a nuance or caveat the technology needs, such as whitelisting files or understanding how the software functions, will get missed. For these reasons, having multiple teams deploy technology will lengthen the amount of time it takes to complete deployment. Consolidating the ownership and responsibilities of technology deployment to a single team will help alleviate this problem. Whether you have a unified deployment team or multiple organizations, you should conduct regular deployment exercises and have a template or playbook to follow to execute deployments. This will ensure consistency in their approaches, communication between all of those involved, and refinement of the process.

Illegal Formation (Aka Lack of Technology Integration)

Each member of a football team has a specific place to stand before a football is snapped.

 

This puts them in the right position to follow the play.

 

Deploying technology during an IR is similar, in that the other technologies in a network need to be in the right position for the deployment to go well. In some cases, organizations do not understand how to configure their security tools, which can block deployment progress. As previously mentioned, failing to whitelist the new technology deployment files with existing antivirus and endpoint security products typically causes deployment impediments. On the network side, misconfigured web proxies and firewalls cause issues when EDR software cannot communicate and send data back to its servers. Specifically, some common problems with network security appliances occur when organizations did not whitelist domains or IP addresses the technology requires for network connectivity.

Coach’s Challenge (Aka CrowdStrike’s Solutions)

To help organizations in the midst of an IR investigation, CrowdStrike offers a Deployment and Integration service where we provide remote and/or onsite assistance in deploying the CrowdStrike Falcon® Platform and Falcon Forensic Collector. Our team will be able to leverage most existing deployment technologies

 

or work with your team to build a custom solution, such as scripting, or leveraging existing solutions, such as PDQ Deploy. In addition, CrowdStrike offers a Live Fire exercise service. This exercise is designed to test individuals within the organization to ensure they understand their roles during an IR scenario. Rather than discuss a hypothetical attack as a group, the team leverages your tools and processes to add to the realism, providing specific information to specific individuals — just as it would occur during an actual breach investigation. The CrowdStrike team then leaves it up to you to determine how best to manage the information. At the conclusion of the engagement, you will quickly understand the weaknesses in your processes, such as network management issues, and gain recommendations and suggestions to improve.

Postgame Wrap-Up

Having an effective technology deployment strategy is imperative to quickly and efficiently scope incident response investigations. Organizations should strive to have a gameplan that maps out multiple strategies for technology deployment in normal and abnormal scenarios e.g. ransomware destroyed the main software deployment application system. Further, having a dedicated, non-siloed deployment team who can follow the deployment gameplan, and integrate the scoping technology with any existing security tools increases the likelihood of success. Finally, conduct a full-speed practice using CrowdStrike’s Live Fire exercise service to put everything to test so you can understand where the gaps are and improve in all aspects of incident response preparedness. Play like a Champion Today!

Additional Resources
Breaches Stop Here
Related Content
CrowdStrike Named a Leader with “Bold Vision” in 2024 Forrester Wave for Cybersecurity Incident Response Services
CrowdStrike Named a Leader with “Bold Vision” in 2024 Forrester Wave for Cybersecurity Incident Response Services
How to Defend Employees and Data as Social Engineering Evolves
How to Defend Employees and Data as Social Engineering Evolves
The Anatomy of an ALPHA SPIDER Ransomware Attack
The Anatomy of an ALPHA SPIDER Ransomware Attack