Securing PostgreSQL from Cryptojacking Campaigns in Kubernetes

  • PostgreSQL misconfiguration allows running shell commands on the underlying OS with the “COPY” SQL command and “pg_execute_server_program” role
  • Kubernetes ingress can allow access to the PostgreSQL service, which can be targeted by the attackers to compromise pods and subsequently a cluster
  • PostgreSQL is targeted by multiple opportunistic cryptojacking groups like WatchDog, TeamTNT and Kinsing
  • The CrowdStrike Falcon® platform helps protect organization of all sizes from sophisticated threats including cryptojacking attempts that exploit PostgreSQL

PostgreSQL is a powerful, open-source relational database management system (RDBMS). Because of its robustness and scalability, PostgreSQL is used extensively in the cloud. Most public cloud providers including AWS, Azure and GCP provide database services to their customers based on PostgreSQL.

According to Google’s January 2023 Threat Horizons report, PostgreSQL is one of the most targeted applications in compromises of Google customers, placing third in frequency (17%) behind SSH (26%) and Jenkins (22%). Weak passwords continue to be the most common vector in the initial access, playing a role in 41% of observed compromises. Attackers are often on the lookout for poorly configured or unsecure PostgreSQL instances deployed in the cloud, Kubernetes or on-premises infrastructure.

The CrowdStrike cloud threat research team previously reported that WatchDog was scanning for PostgreSQL. Many opportunistic cryptojacking threat actors, including TeamTNT and Kinsing, have targeted misconfigured PostgreSQL instances to mine cryptocurrency. This blog takes a look at the most popular misconfigurations exploited by these adversaries.

Take the CrowdStrike Cloud Security Challenge with a free Cloud Security Health Check.

PostgreSQL Misconfiguration Allows Cryptojacking

Common PostgreSQL services hosted by cloud service providers (such as Amazon RDS for PostgreSQL, Azure Database for PostgreSQL or GCP Cloud SQL for PostgreSQL) do not allow customization of authentication controls that could lead to this abuse, but when PostgreSQL is deployed manually, many misconfigurations in authentication methods, user roles, access and permissions could occur that may create exploitable exposures.

 

PostgreSQL supports multiple authentication methods when deployed manually. Of them, “trust” authentication is the most insecure and provides access without the need for any password, which is likely why cryptomining groups like to target PostgreSQL cloud instances with authentication type “trust” enabled. There are two ways in which “trust” authentication can be enabled:

 

  1. Using a pg_hba.conf file
  2. Using the environment variable POSTGRES_HOST_AUTH_METHOD=trust

Figure 1 shows “trust” authentication using pg_hba.conf where any user from any IP can connect to the PostgreSQL server without a password. The username could be a superuser name, in which case the attacker would inherit all of the privileges of a PostgreSQL superuser.

Figure 1. Insecure pg_hba.conf file

“Trust” authentication can also be enabled by setting the environment variable POSTGRES_HOST_AUTH_METHOD=trust inside a Kubernetes pod or a server. Once set, any IP or user can connect to the database without a password. Additionally, “trust” authentication combined with risky PostgreSQL user default roles (noted below) provides a perfect opportunity for attackers to abuse databases to read or write sensitive files on the filesystem, modify the configuration and escalate privileges.

  1. pg_read_server_files (allows OS file reads)
  2. pg_write_server_files (allows OS file writes)
  3. pg_execute_server_program (allows OS binary to execute)

In an attack on misconfigured PostgreSQL, a remote attacker can authenticate as a superuser or user having the right roles — e.g., pg_execute_server_program enables them to download and run suspicious files to mine cryptocurrency. Figure 2.A shows attackers using the COPY command to download and run xmrig, which mines digital currency like Monero; figure 2.B shows the xmrig process running inside a container successfully mining the currency.

Figure 2.A. An attacker using the COPY command to download and run xmrig
Figure 2.B. Xmrig mining cryptocurrency by remotely accessing misconfigured PostgreSQL

As another example, Figures 3.A and 3.B show how misconfigured PostgreSQL can be used to list any sensitive file without the use of any suspicious tools. Once an attacker has a foothold into the PostgreSQL service, they can set cron jobs to get a reverse shell, escalate privileges to the container root and pivot to attack the Kubernetes host, API or cloud provider’s API to escalate and take over a Kubernetes cluster or cloud account.

Figure 3.A. An attacker using the COPY command to read sensitive files in /etc/passwd
Figure 3.B. The results of an attacker using the COPY command to read sensitive files

CrowdStrike Detection and Protection

The Falcon platform unifies cloud security to deliver comprehensive protection to its customers from any attacks on Kubernetes, cloud or hybrid infrastructure. The Falcon platform, with its machine learning models, detects and prevents runtime threats to the cloud, Kubernetes or serverless infrastructure. In the example shown in Figure 4.A, the Falcon platform identifies and detects the child process of PostgreSQL trying to download and launch the xmrig miner in a container. Figure 4.B shows the Falcon platform identifying and killing the malicious process early in the kill chain when a malicious file was being downloaded. Figure 4.C shows Falcon platform detecting PostgreSQL mis-configuration at runtime.

Figure 4.A. The Falcon platform detecting and alerting on each stage of the kill chain
Figure 4.B. The Falcon platform killing the wget process that is used to download malware
Figure 4.C. The Falcon platform detecting an insecure PostgreSQL configuration using the environment variable (click to enlarge)

The Falcon platform takes a defense-in-depth approach to protecting customers by leveraging incoming telemetry to power detection and provide real-time mitigation. It includes the following indicators, which are used to detect cryptojacking attempts like the ones discussed:

  1. Container drift prevention
  2. Rogue container running on your Docker instance
  3. Misconfigured Kubernetes or Docker instance

Public cloud managed services for PostgreSQL are not vulnerable to this misconfiguration. CrowdStrike Falcon® Horizon cloud security posture management enables DevOps practitioners to keep applications like PostgreSQL secure by proactively monitoring, identifying misconfigurations and ensuring compliance in the public cloud infrastructure. The following table is a list of Falcon Horizon indicators of misconfigurations (IOMs) for PostgreSQL that help identify security gaps.

PlatformServiceSeverityPolicy NamePolicy Description
AzurePostgreSQLMediumPostgreSQL database does not enforce SSLA PostgreSQL database server was identified that does not enforce SSL for connections to the database. This means clients may establish cleartext connections to the database, allowing their traffic to be monitored/intercepted if they are connecting over untrusted networks.
AzurePostgreSQLInformationalPostgreSQL database has the connection throttling parameter disabledA PostgreSQL database with the connection throttling parameter set to “off” was identified. Connection throttling will temporarily throttle inbound connections from an IP address after several failed login attempts, protecting the database from brute force attacks.
AzurePostgreSQLInformationalPostgreSQL database has the log checkpoints parameter disabledA PostgreSQL database with the checkpoint logging parameter set to “off” was identified. Without this type of logging, you will not have insight into the checkpoint process, which may be disruptive to database performance and connection reliability.
AzurePostgreSQLInformationalPostgreSQL database has the log connections parameter disabledA PostgreSQL database with the connection logging parameter set to “off” was identified. Without this type of logging, you will not have insight into connection attempts made to the database. Connection logs can help identify when brute force attacks are being attempted against a database and can assist with tracking down adversaries during an investigation.
AzurePostgreSQLInformationalPostgreSQL database has the log disconnections parameter disabledA PostgreSQL database with the disconnection logging parameter set to “off” was identified. Without this type of logging, you will not have insight into connection sessions with the database, which may include details such as session length and disconnection time.
AzurePostgreSQLInformationalPostgreSQL database has the log duration parameter disabledA PostgreSQL database with the duration logging parameter set to “off” was identified. Without this type of logging, you will not have insight into the time it takes to complete queries against the database. This may aid in identifying when the database is under a stressful load or if there are other performance issues with the queries and/or database.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_checkpoints” flag is disabledEnabling log_checkpoints causes checkpoints and restart points to be logged in the server log. Some statistics are included in the log messages, including the number of buffers written and the time spent writing them. This parameter can only be set in the postgresql.conf file or on the server command line. This recommendation is applicable to PostgreSQL database instances.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_connections” flag is disabledPostgreSQL does not log attempted connections by default. Enabling the log_connections setting will create log entries for each attempted connection as well as successful completion of client authentication, which can be useful in troubleshooting issues and to determine any unusual connection attempts to the server. This recommendation is applicable to PostgreSQL database instances.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_disconnections” flag is disabledPostgreSQL does not log session details such as duration and session end by default. Enabling the log_disconnections setting will create log entries at the end of each session, which can be useful in troubleshooting issues and to determine any unusual activity across a time period. The log_disconnections and log_connections work hand-in-hand and generally, the pair would be enabled/disabled together. This recommendation is applicable to PostgreSQL database instances.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_lock_waits” flag is disabledThe deadlock timeout defines the time to wait on a lock before checking for any conditions. Frequent runovers on deadlock timeout can be an indication of an underlying issue. Logging such waits on locks by enabling the log_lock_waits flag can be used to identify poor performance due to locking delays or if a specially crafted SQL is attempting to starve resources through holding locks for excessive amounts of time. This recommendation is applicable to PostgreSQL database instances.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_temp_files” flag is disabledThe log_temp_files flag is disabled. Temporary files are not logged so it may be more difficult to identify potential performance issues that may be due to deliberate resource starvation attempts.
GCPCloud SQLInformationalCloud SQL for PostgreSQL “log_min_duration_statement” flag is enabledLogging SQL statements may include sensitive information that should not be recorded in logs. This recommendation is applicable to PostgreSQL database instances.
GCPCloud SQLMediumCloud SQL database instance does not require SSLSQL database connections if successfully trapped (MITM) can reveal sensitive data like credentials, database queries, query outputs, etc. For security, it is recommended to always use SSL encryption when connecting to your instance. This recommendation is applicable for PostgreSQL, MySQL generation 1 and MySQL generation 2 instances.
GCPCloud SQLInformationalCloud SQL instance not configured for backupsBackups provide a way to restore a Cloud SQL instance to recover lost data or recover from a problem with that instance. Automated backups need to be set for any instance that contains data that should be protected from loss or damage. This recommendation is applicable for SQL Server, PostgreSQL, MySQL generation 1 and MySQL generation 2 instances.
GCPVPCHighVPC allows access to PostgreSQL from all networksAllowing ingress network traffic from the global IP space (0.0.0.0/0) exposes unnecessary, commonly exploited services to any attacker on the internet. This high-risk port is commonly scanned for by attackers, and a single misconfiguration or out-of-date server could pose a serious risk to the organization. The access gained by the attacker could result in the compromise of data or further exploitation in the network hosting the exposed services.

MITRE for Defending Infrastructure as a Service (IaaS)

Crowdstrike has teamed up with MITRE Engenuity Center for Threat-Informed Defense to develop ATT&CK Defense for IaaS. This project’s goal was to develop effective methods and strategies for defending against adversaries attacking IaaS across the cloud, containers and Linux aaS. Because the most popular technique used by attackers to get initial access into cloud infrastructure is to exploit a public-facing application like postgreSQL, it’s paramount to assess the internet exposure of your enterprise application and subsequently follow best practices to secure them.

Best Practices for Securing PostgreSQL

Securing a PostgreSQL workload requires a combination of good configuration practices and regular monitoring. Here are some general guidelines for securely deploying PostgreSQL in cloud and Kubernetes environments:

  1. Use the latest version of PostgreSQL and install necessary patches.
  2. Use a strong password when using password-based authentication methods.
  3. Apply appropriate permission to secure configuration files like pg_hba.conf.
  4. Enable SSL/TLS to secure the connection between the client and the PostgreSQL service.
  5. Audit users and their security roles and follow the principle of least privilege.
  6. Use Kubernetes namespace’d secrets.
  7. Run the PostgreSQL service as a non-root user, which will add an additional layer of defense in case of compromise.
  8. Limit the container resources available to the PostgreSQL service to an estimate.
  9. Monitor hosts and containers for any malicious activity.
  10. Use a Zero Trust mechanism to allow required access to the service in the cluster.
  11. Use a proactive security solution to identify misconfiguration and vulnerabilities

Conclusion

Securing PostgreSQL, a popular cloud-based application, is crucial for preventing attacks on your infrastructure. Misconfigurations in PostgreSQL can serve as an entry point for attackers, as seen in cases where cryptojacking groups have taken advantage of such vulnerabilities to mine cryptocurrency for profit. To protect against such breaches, it is important to follow best practices for securing PostgreSQL, actively monitor for misconfigurations and maintain a strong overall security posture.

One of the ways to achieve this is by using a comprehensive cloud security platform such as CrowdStrike’s adversary-focused cloud-native application protection platform (CNAPP), which offers a combination of runtime security and proactive measures, including assessments for IOMs and indicators of attack (IOAs) as well as compliance checks. It also detects and prevents breaches by various types of actors, including cryptojacking hacktivists, eCrime groups and nation-state actors, providing a complete and robust solution for securing your cloud infrastructure. Schedule a demo to see CrowdStrike’s CNAPP solution in action: Cloud Security Posture Management | Cloud Workload Protection | Container Security | Cloud Infrastructure Entitlement Management

Additional Resources