CrowdStrike Research Challenges Containerized Application Predictability Assumptions

• General-purpose Linux utilities — often observed in living off the land post-exploitation activity — are frequently used in popular containerized applications.
• CrowdStrike researchers have identified significant variation in the usage patterns of general-purpose Linux utilities across these containerized applications.
• The findings challenge commonly accepted assumptions about the predictability and immutability of containerized workloads.
• Cybersecurity solutions that solely rely on the perceived predictability and immutability of containerized workloads are ineffective.

Cloud workloads — and containers in particular — are often seen as immutable entities with predictable behavior. But recent CrowdStrike research suggests that some cloud security solutions rely too much on this premise, leading to suboptimal detection outcomes.

CrowdStrike observes billions of container events each day. The data we collect gives us insights into real-world cloud workload behavior, which challenges these assumptions. In particular, our research demonstrates that Linux utilities often used as living off the land (LOTL) post-exploitation tools are also commonly seen in containers, including widely used containerized applications. In most cases, their usage is not a sign of malicious behavior. 

In this post, we’ll share the results of our research — as well as our interpretation and intuition behind it — and show why a detection approach that solely relies on the perceived predictability of containerized workloads is ineffective.

Are Containerized Applications as Predictable as We Expect Them to Be?

This is the question we set out to answer. To narrow the scope, we focused on well-known containerized applications, including various databases, message brokers, orchestrators, automation servers and web servers, among others. The wide adoption of these applications, along with the many reported vulnerabilities and exploits affecting them, makes them an appealing attacker target. 

Here is the premise we tested: If the core functionality, behavior, configuration options and use cases of an application running in a container are well understood and predictable, it should be possible to detect deviations from the expected behavior and use these detections to identify malicious activity in the container. To narrow the scope even further, we selected applications that aren’t heavily customized by users and therefore shouldn’t demonstrate vastly different behavior across the environments they run in. 

A typical containerized application that we looked at:

• Typically runs in a dedicated single-purpose container
• Typically has several instances running within a company’s environment
• Is widely used across the industry
• Limits user customization options

Using this approach, we selected the following applications for our in-depth analysis: 

• Elasticsearch
• Kafka
• MongoDB
• MySQL
• Nginx
• PostgreSQL
• RabbitMQ
• Redis
• ZooKeeper

Living Off the Land

The post-exploitation phases of an attack often leverage popular Linux utilities that are readily available in containers. These LOTL techniques are not new. We wanted to see how often these utilities are run within our set of selected applications and whether these occurrences can be used as a reliable indicator of a malicious activity in a container where an application is running. 

We selected the following set of common Linux utilities to focus on for this analysis:

Data-Driven Insights

CrowdStrike regularly processes trillions of data points daily, enabling our data scientists to run large-scale analyses and uncover valuable information. Our analysis of billions of container events across a large set of diverse companies yielded a few insights.

First, Linux utilities are widely used in containers in general. For close to two-thirds of the companies analyzed, we identified at least one of these utilities running within a container in their environment. Further, we counted the number of different utilities seen running in containers, broken down by application container in focus for this analysis. Figure 1 shows the results, sorted from highest to lowest:

At the high end, we saw 29 different utilities running across all Nginx containers. With the exception of RabbitMQ, the rest of the applications show similar results, having at least 24 observed different Linux utilities. This is a high number of unique utilities seen per application, as there are only 34 different utilities in focus for this analysis.

The high number of different Linux utilities observed is somewhat surprising if it’s assumed that these containerized applications should have predictable and consistent usage across different  environments. The results also demonstrate that these Linux utilities, which are often used as LOTL post-exploitation tools, are commonly seen in normal container behavior.

To further understand the usage of application containers, a more comprehensive metric was needed to effectively capture the notion of similarity. For this, we used Jaccard similarity. The Jaccard similarity index is defined as the size of the intersection divided by the size of the union of the sample sets. Mathematically, for two sets A and B:

Jaccard similarity helps calculate similarity across a set of objects because it effectively measures the proportion of shared attributes between objects relative to their combined unique attributes. Jaccard similarity is also easy to understand, intuitive and explainable.

Let’s look at a simple example to make it more clear. Let’s assume you have three containers with the following sets of observed Linux utilities:

The resulting pairwise Jaccard similarities would be:

​We can then compute the average Jaccard similarity (equivalent to percentage overlap) across all pairs to get a single metric for overall similarity:

From the percentage overlap metric calculated above, we would say that on average, 38% of the Linux utilities overlap across the different containers. This means that though there’s some commonality, there’s also significant diversity in the Linux utilities used across containers.

Now that we have a clear methodology, we can apply this to the containers of each application in our analysis. Figure 2 provides the calculated average percentage overlap metric for each application, sorted from lowest to highest:

We observe only a 29% average overlap of Linux utilities for ZooKeeper containers and 37% for Elasticsearch. The highest overlap was MySQL (76%), which still is relatively low in this context. 

For a strong common profile, we would expect to see percentages closer to 100%. The relatively low percentage of overlap calculated above shows that there’s significant diversity in the Linux utilities used across these containers.

Even within a given company's environment, we observed large variations in the set of Linux utilities seen among containers, with hundreds of companies having per-application average overlap percentages under 50%. This further demonstrates that there is significant diversity in Linux utilities used across containers, even for common third-party application containers within a given company’s environment. 

Based on our analysis, it’s clear that establishing accurate container profiles based on Linux utility usage alone is impossible. 

Typical Use Cases for Linux Utilities in Containers

Digging deeper, we looked at the reasons for wide use of the standard Linux utilities. We found that it can be attributed to the following activities commonly seen in containers. The following activities were performed by either custom scripts included in the container images or by exec’ing into the running instances of these containers:

• Periodic health status checks and heartbeat activity that typically involves creating a cronjob using crontab and periodic ping or http requests using curl.
• Periodic metric export activity that typically involves posting matrices using https requests using curl and occasionally compressing the metric data using tar/gzip.
• Periodic data backup activity that typically involves creating a cronjob using crontab for scheduling backups, compressing the data using tar/gzip and pushing it to the backup systems using curl.
• Use case-specific bootstrap scripts that perform a wide variety of activities, including security setup using iptables, configuration and credential provisioning by accessing secret vaults and identity management systems using curl, neighborhood connectivity checks using ping/nc/route, fetching and installing external initialization scripts and utilities using curl/wget/gzip/apt/rpm/gpg and environment checks using printenv.
• There are also one-off cases that don’t follow any known patterns and that include administrative and development activities.

These observations led us to the following conclusions:

1. Linux utilities are used frequently in containers for benign purposes, and they are included in container images for this reason.
2. Every environment is different, and companies typically have more than one distinct environment. This means that there’s high variation in usage patterns for standard Linux utilities between companies and between environments within the same company.
3. This variability is observed for the same application types as well. Even though some of the usage patterns are periodic in nature, high variability makes it challenging to differentiate between expected and unexpected use of Linux utilities.

Detection Dilemma: Use of Linux Utilities in Containers Isn’t Always a Sign of Post-Exploitation Activities

Our research demonstrates that Linux utilities are widely used in application containers and are not a reliable sign of malicious activity. A detection approach that only focuses on the usage of post-exploitation utilities will be unsuccessful. 

CrowdStrike’s approach to cloud security includes the full suite of cloud-native application protection platform (CNAPP) capabilities and a comprehensive approach to detecting cloud threats. CrowdStrike’s rich and large dataset enables our data scientists and detection developers to gain key insights into modern cloud environments and identify the best combination of methods to protect against attacks on cloud workloads and infrastructure. 

Work like this drives innovation. By publishing their findings on this blog and in journals, CrowdStrike researchers and data scientists not only ensure that the CrowdStrike Falcon® platform is the best cybersecurity technology on the market but help advance the cybersecurity industry in general.    

Additional Resources

• Learn more about the Falcon platform by visiting the product webpage.
• Learn more about CrowdStrike endpoint detection and response by visiting the CrowdStrike Falcon® Insight XDR webpage.
• Test CrowdStrike next-gen antivirus (AV) for yourself. Start your free trial of CrowdStrike® Falcon Prevent™ next-gen AV today.