As containerization technology gains popularity, Docker has emerged as a leading solution for building containerized applications. To optimize the usefulness of Docker logs, it is critical to understand how they are produced and handled.
In this article, we’ll explore advanced topics related to Docker logging, focusing on logging drivers and the different ways to produce logs. We’ll also discuss Docker logging delivery modes and how these affect log processing to a driver.
Logging drivers
Containers generate Docker container logs and therefore need a way to be collected. To facilitate log collection, Docker provides logging drivers. These are plugins that allow Docker to route container logs to various destinations and offer different mechanisms to retrieve container logs.
Docker offers a range of logging drivers, including Syslog, Journald, and Fluentd. Each logging driver has its own configuration options to support different use cases and logging destinations. The json-file driver is the default log driver.
To configure Docker logging drivers, either modify the Docker daemon configuration file or specify the logging driver using the --log-driver option when creating a container. The configuration file allows users to set the default logging driver for all containers, while the --log-driver option lets users specify the --log-driver for a specific container.
To set Syslog as the default logging driver for all containers, you would use the following in your configuration file:
{"log-driver": "syslog"
}
Alternatively, specify the logging driver using the --log-driver option when starting a container, as per the example below:
~ docker run --log-driver syslog my-app
Dual logging
Dual logging refers to using the docker logs command consistently, regardless of the configured logging driver. However, when configuring Docker’s logging drivers, you can disable dual logging in the configuration file, as shown below:
{"log-driver": "syslog",
"log-opts": {
"cache-disabled": "true",
}
}
With dual logging disabled, you will encounter an error when attempting to read container logs locally. For example, suppose you have used the following code to configure a remote logging driver for a container but have disabled dual logging.
~ docker run --log-driver syslog --log-opt syslog-address=udp://my-logging-server:514 my-app
Then, by attempting to read the logs locally using the docker logs command, an error message will appear similar to the following:
Error response from daemon: local logging endpoint is not enabled while using remote logging driver.
Output customization
In addition to configuring logging drivers, Docker also provides customization options for the output of container logs by using the tag option in the --log-opt flag. Using this option, you can add metadata to the log messages produced by the container. With metadata attached to log messages, you can filter and route log messages to different destinations.
For example, assume you have a container that is running a web server, and you want to tag all log messages produced by that container with the server’s name. You would use the tag option like this:
~ docker run --log-driver=json-file --log-opt tag=web-server-1 my-app
This tag will start a container running the my-app image, and tags all log messages produced by the container with the value web-server-1.
Consider another example, where there are multiple containers running the same application. In this scenario, you want to differentiate between log messages produced by different containers. You might use the container ID as the value for the tag option, as below:
~ docker run --log-driver=syslog --log-opt tag="{{.ID}}" my-appThis tag option will start a container running the my-app image, but all log messages the container produces will be tagged with the container’s ID.
Note that the syntax for specifying the tag value differs for each logging driver. For example, when using the Syslog driver, you would use the Go template syntax to specify the tag value. Refer to the documentation to learn more about the syntax for specifying the tag value for the logging driver.
Docker logging delivery modes
In Docker, the delivery mode refers to how logs are delivered from the application running in a container to the logging driver configured for that container. There are two primary delivery modes in Docker: direct/blocking and non-blocking.
Direct/blocking
The direct/blocking delivery mode is the default mode in Docker. In this mode, the application sends logs directly to the logging driver, interrupting the application until log dispatch completion. This guarantees that all logs will be sent. However, since the application must wait for the logging driver to process and store the logs before continuing its execution, this delivery mode can cause latency.
Syslog and Journald are examples of logging drivers that use the direct/blocking delivery mode.
Non-blocking
The non-blocking delivery mode writes logs to an in-memory ring buffer, which is a fixed-size circular buffer in memory. In this mode, the logging driver processes the logs when the ring buffer is full. Since the application does not have to wait for the logging driver to process the logs, this mode can reduce latency. However, if the ring buffer fills up before the logging driver completes processing the logs, this can result in log loss.
Fluentd and json-file are examples of logging drivers that use this delivery mode.
The choice of delivery mode will depend on your specific use case and application requirements. For example, if you need real-time visibility into your application's logs and don't mind the potential for delays or failures, a direct/blocking delivery mode may be suitable. However, a non-blocking delivery mode may be more appropriate if high performance and availability are more important.
Configure delivery modes
Docker allows the configuration of delivery mode in different ways, either through its configuration file or the --log-opt option when starting a container.
The direct/blocking delivery mode is the default, so no changes are necessary for this configuration. However, to change the default and configure the non-blocking delivery mode in the config file, you can set the logging driver options in the file as follows:
{"log-driver": "json-file",
"log-opts": {
"mode": "non-blocking"
}
}
Alternatively, you can specify the non-blocking delivery mode using the --log-opt option when starting a container as per below:
~ docker run -it --log-driver=json-file --log-opt mode=non-blocking <image_name>
Summary
Logging drivers are an important aspect of Docker logging. They provide different mechanisms for retrieving logs and make it easier to analyze and troubleshoot issues. Docker also allows for the configuration and customization of Docker logging driver output and leverages the tag option to add metadata to log messages.
The delivery mode is a key consideration when selecting a logging driver, but which delivery mode to use will depend on your application’s needs. Configuring delivery modes through the configuration file or the --log-opt option helps you achieve optimal log delivery methods and improves logging for your application requirements.
Log your data with CrowdStrike Falcon Next-Gen SIEM
Elevate your cybersecurity with the CrowdStrike Falcon® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.
