Automating Remote Remediation of TrickBot via Falcon’s Real Time Response API: Part 2

July 09, 2020

| | From The Front Lines
This is Part 2 in a two-part blog series covering the CrowdStrike® Falcon Complete™ team’s ability to remotely remediate “TrickBot,” a modular trojan that is particularly devastating when paired with “Ryuk” ransomware. This deep dive analyzes an automated methodology that leverages the Falcon Real Time Response (RTR) API in addition to PowerShell and Python scripting in order to remotely remediate TrickBot infections at scale. As the global workforce continues to transition to remote work from home, it is increasingly critical for defenders to leverage tools with remote detection and remediation capabilities. The remediation of dozens or hundreds of hosts individually is not practical or cost effective for widespread infections — and this is especially true when the systems are widely distributed and not concentrated in a small number of locations or facilities. A recovery approach that combines remote access with an automated methodology provides a powerful alternative to address the limitations of manual remediation and/or rebuilding systems on an individual basis.

 

TrickBot Analysis and Remediation Overview

 

The adversary tracked as WIZARD SPIDER has used “big game hunting” tactics to great effect against a diverse set of targets. TrickBot is often delivered to endpoints via email phishing campaigns and deployed as a secondary payload. In addition to credential theft and wire fraud, TrickBot also possesses modular capabilities that allow it to spread widely throughout a network through exploitation of SMB (Server Message Block), brute forcing RDP (Remote Desktop Protocol) or via network shares. Once WIZARD SPIDER has gained a foothold within an environment with successful TrickBot infections, it may then pivot to manually deploying Ryuk payloads and begin ransoming hosts. Ryuk has impacted organizations in multiple industry verticals, including healthcare, government, education and the private sector. We presented an illustration of the remote remediation process in Part One, which covered the basic steps required to complete the process manually via the CrowdStrike Falcon®® RTR console. In Part Two, we explain the automated methodology. Mass remediation of TrickBot via an automated script process saves a significant amount of time and resources that can be better allocated for other risk-reducing initiatives or priority recovery tasks.

Putting It All Together and Hitting the “Easy Button”: Leveraging the RTR API and Scripts for Automated TrickBot Remediation

The automation aspect of this overall approach can be broken into three distinct requirements:
  1. Configuring the API client
  2. Developing PowerShell scripts for remediation
  3. Developing Python script for API integration
First, the CrowdStrike API is a set of REST-based API endpoints that allow you to perform actions programmatically instead of via the Falcon console. This is what allows us to not only remediate hosts remotely, but to do it at scale. Next, the RTR runscript feature of the Falcon agent can be leveraged to easily create and save PowerShell scripts, so that they can be staged to run across a network environment. In lieu of running individual PowerShell one-liners as shown in Part 1 of the blog series, they can be chained into functions of a larger script to increase efficiency. Finally, a Python script can be leveraged to interact with the RTR API in order to deploy the staged PowerShell to conduct remediations across a range of targeted hosts in an environment.

 

Accessing and Configuring the API Client

CrowdStrike offers many API endpoints. Falcon users can find documentation and sample use cases from within the Falcon console. Specific details regarding how to access and configure the API client are omitted here since they are out of scope. Falcon customers should reach out to their account managers for more information on the API endpoints. For the purposes of this use-case example, the discussion proceeds assuming a user has successfully obtained an access token, generated an API client/key, and applied the appropriate permissions for RTR functionality.

 

The PowerShell Scripts

Falcon administrators can create and upload PowerShell scripts to the Falcon Console. CrowdStrike recommends the utilization of two PowerShell scripts with this approach to minimize the potential for errors and allow the opportunity to review output prior to modification of the file system on a remote host. By using a “query first, then kill” methodology, users can confirm that a host is infected, and that only residual artifacts related to the malicious activity are removed prior to running any remediation scripts. The initial query script performs no invasive actions, and simply queries for signs of infection, then outputs results. During development, the output should be reviewed by an analyst to ensure the script’s efficacy. This helps to ensure that no legitimate system resources are stopped or removed. The query script for this case study is designed to identify TrickBot artifacts (services, scheduled tasks, directories, binaries, etc.). The script must be updated and modified to include new indicators that are identified via triage during an incident investigation. For example, the installation directory locations under *\AppData\Roaming is not a reliable indicator over time as these naming conventions are variable and change frequently between different TrickBot campaigns.

 

 The CrowdStrike Falcon® Complete team’s recent experience suggests that a small number of hosts should be manually triaged by analysts in order to identify and enumerate the observed IOCs and/or TTPs for a particular TrickBot campaign. This data can be used to build out the flexible modules of the script’s functions. Due to the dynamic nature of the threat and variance between samples, there is no specific number of hosts on which this must be performed.

 

Processes

The function shown in Figure 1 below is part of the kill script that automates looping through the malicious processes and terminating them automatically. To successfully accomplish this and remove the correct processes and avoid killing legitimate processes, the output from the TASKLIST is modified with a SPLIT function and a regular expression pattern to match the data for the correct process IDs.
Figure 1. Identifying and Terminating Malicious svchost.exe PIDs
Once the malicious processes are killed, the next remediation stage is to identify and remove any persistence mechanisms that have been established by the malware. In this scenario, two different persistence tactics are identified and handled in turn.

Persistence (Scheduled Tasks & Services)

The function in Figure 2 shown below is part of the kill script that automates the removal of scheduled tasks with known task names. The scheduled task names that are used in the Where-Object statement are IOCs that must be identified by manual triage, and then simply plugged into the script. In contrast to the manual steps shown in Part 1 of this blog series, the timing of the deletion is a non-issue, because the scripted deletion will take care of the persistence immediately after the process is terminated in the previous step.
Figure 2. Querying for and Deleting the Scheduled Task Automatically
In addition to scheduled tasks, TrickBot will often start a new service to establish multiple persistence mechanisms. The function shown below in Figure 3 is part of the script that automates the querying for services to match a regular expression pattern for known TrickBot naming conventions and/or TTPs identified in triage.
Figure 3. Scanning for and Deleting Malicious Services (Click image to enlarge)
After the persistence mechanisms via both scheduled tasks and services have been successfully removed, the final step in the remediation process is the clean-up of any residual artifacts in the files and folders remaining on-disk.

 

Remaining Files/Folder Artifacts

Figure 4 shown below is part of the script that automates the querying for known locations that likely contain a TrickBot related binary, and then removing the malicious files. One additional note on the automated script is that it contains a function designed to search for binaries in the GPO cache directories for the local user and local machine, as CrowdStrike has observed TrickBot binaries staged in these locations in recent incidents. This is an uncommon TTP, one that followed the compromise of a Domain Controller which was used to push out TrickBot via a group policy object (GPO).

 
Figure 4. Script Excerpt to Query and Remove Malicious TrickBot Files and Folders (Click image to enlarge)

Remediation Output

 

The Remediation Script is designed to identify TrickBot artifacts (services, scheduled tasks, directories, binaries, etc.) and then remove them from the infected host. The script has multiple try/catch blocks and ForEach loops to remove the TrickBot artifacts that it identifies. Again, for careful and precise remediations, a query/kill methodology should be used to first identify false positives.

 

 The script as presented in these examples will only display the services, tasks, and files it finds. It is best practice to configure the script so that headers such as “No Trickbot artifacts found” or “” as the first line of output. The header allows for quick review of the CSV output via filters.

 

 

Practical Application with Python

The python script utilized in this process is another custom script that must be developed in order to interact with the RTR API. This script is the driver that handles the input/output, API calls, and also points to the specified PowerShell scripts staged in Falcon. It is executed from the operator’s local host. The python script requires the following:
  • python3
  • python3 requests library
CrowdStrike suggests putting the script in a folder by itself with the name, mass-rtr.py. Also, before executing the script, Falcon Complete recommends creating a CSV file named hosts_to_execute.csv in the same folder. This file contains the hosts that will be targeted for remediation, although users could in theory choose to configure the script to target hosts in another way. The unique hostnames and corresponding AID’s can be obtained by performing an Event Search query that searches for known indicators and identifies infected hosts. The hosts_to_execute.csv file itself will hold some necessary input fields for the Python script. This file’s format is shown in Figure 5 below. The following outlines the overall logical process.

 
  1. Read in CSV of hosts to execute script on with Headers: hostname, aid, enabled
    • hostname is the device’s hostname for human readability. This value will not be used by the code other than to put into the output file.
    • aid is the device ID you can find within the Falcon Console.
    • enabled can be TRUE or FALSE. You can use this as a toggle switch to execute on certain host(s) or not.
Figure 5. Snippet of CSV File hosts_to_execute.csv
Users will need to add a few test hosts to hosts_to_execute.csv navigate to the Falcon Console to create a custom script to run. A good starting script would be: ps | out-string. This command will display all the running processes on the system. Some commands using RUNSCRIPT are represented differently in standard output (stdout). Thus, running | out-string at the end of each powershell command is a good idea to normalize your output. Once testing is completed with a starting script, users should be able to add the more complex targeted scripts, for example the Trickbot Remediation script in this case. An example of a custom script is shown in Figure 6.
Figure 6. Custom Script Example (Click image to enlarge)
Users will need to create python code once the API client and PowerShell script have been made. For example, on lines 20 - 22 in Figure 7, users will need to customize values to their own environment.
Figure 7. Sample Code To Run Script
As noted above in the section for Accessing and Configuring the API Client, the script will need to take three steps with the API for this remote remediation process. Finally, the script must include a way to handle the output which is previewed in Figure 8. Below are some snippets of the Python code that users will need to develop.

 

1. Get an access token as shown in Figure 8.

Figure 8. Obtain Access Token

2. Connect to hosts as shown in Figure 9:

Figure 9. Establish Host Connection (Click image to enlarge)

3. Execute a script on hosts as shown in Figure 10:

Figure 10. Remote Script Execution (Click image to enlarge)

4. Falcon Complete chooses to log output with the following format: a CSV with each row being a different host with headers: hostname, aid, stdout, stderr, offline as shown in Figure 11. This adds a desired capability to our workflow for ticketing and documentation purposes, but there are many other ways to develop the script and adjust the output for any users’ requirements.

    • hostname is the device’s hostname for human readability
    • aid is the device ID you can find within the Falcon Console
    • stdout is the standard output that the API returns for the specific host. This comes from the stdout of the script running on the host.
    • stderr is the standard error that the API returns for the specific host
    • offline will be TRUE if the device is not online, FALSE if the device is online or "?" if any exception occurred when attempting to read the output.
Figure 11. Recording Output
Now that hosts_to_execute.csv is populated, the PowerShell script is created, the API client is created, and the values in the script are populated in the script, users will be able to run the test script. Users can run the test script with the following command: python3 mass-rtr.py. An example run is shown in Figure 12.
Figure 12. Example Script Run
After execution of the script there will be a newly created file called output.csv. It will look like something similar to the output shown in Figure 13.
Figure 13. Example Output
When the script completes, the infected hosts have been remediated via the remote session. The script output should then be reviewed for efficacy and it is recommended that the output is retained according to the user’s document retention policy.

 

Conclusions

 

This was the final part of a 2-part blog series that covered the Falcon Complete team’s capability to remotely remediate “TrickBot” via an automated methodology that leveraged the Falcon Real Time Response (RTR) API in addition to PowerShell and Python scripting in order to remotely remediate TrickBot infections at scale.

 

 CrowdStrike has observed multiple instances where banking malware and follow-on ransomware had a highly disruptive impact on organizations’ ability to operate. Remediating these types of infections becomes more complicated with these variants’ ability to spread to many hosts. Further, with a transition to a more remote workforce, the capability to remotely remediate infections on hosts that are geographically distributed will become increasingly important as rebuilding systems becomes impractical. A recovery strategy that restores hosts without rebuilds can save critical resources in times of crisis so that organizations can get back on their feet as quickly as possible.

 

Recommendations

Additional Resources
Breaches Stop Here
Related Content
CrowdStrike Named a Leader with “Bold Vision” in 2024 Forrester Wave for Cybersecurity Incident Response Services
CrowdStrike Named a Leader with “Bold Vision” in 2024 Forrester Wave for Cybersecurity Incident Response Services
How to Defend Employees and Data as Social Engineering Evolves
How to Defend Employees and Data as Social Engineering Evolves
The Anatomy of an ALPHA SPIDER Ransomware Attack
The Anatomy of an ALPHA SPIDER Ransomware Attack