In part one of this series, we covered how syslog works, the syslog message format, and the components of a syslog server. We also discussed some pros and cons of using syslog for collecting and analyzing events.

Here in part two, we’ll dive deeper into the configuration of syslog to show you how to get the most out of your syslog logging setup. We’ll learn how to configure syslog within the rsyslog.conf file and how to manage log rotation. Finally, we’ll cover some best practices for using syslog in log collection and analysis.

Understanding the Syslog Configuration File

The rsyslog.conf file determines how the syslog server handles log messages. It contains a list of rule statements that define which messages to match and what actions to take. Rsyslog is the server process daemon used on most Linux distributions for processing logs in the syslog format.

Each time rsyslog receives a message, it scans through the configuration file to check if it matches any of the rules. If it does, rsyslog takes the configured action. If not, rsyslog discards the message.

Selectors

Selectors consist of the facility and priority used to compare each message against, separated by a period.

As we noted in part one of this series, facilities specify which program generated the event (for example: auth, cron, daemon, kern, or user). You can create multiple lines for different facilities to take different actions depending on the program.

Priorities determine the criticality of the event (such as debug, info, error, critical, alert, or emergency). Priority selectors are hierarchical, matching the specified priority and all higher priorities. For example, if critical is specified, then syslog will match alert and emergency priority messages too.

In this example, the selector is made up of the “daemon” facility and the “notice” priority. Any messages that match this facility and priority (or higher) will be logged to the /var/log/daemon.log file.

daemon.notice /var/log/daemon.log

Actions

Actions determine how syslog handles messages. You can carry out multiple actions on a single message. Let’s examine possible actions in detail.

Specify a file location: Use this to specify which file location syslog should save messages to. This can be a static name, or it can be dynamic, based on the date or the hostname, or some other value. This helps organize syslog messages so that they are easier to retrieve for analysis.

The following example saves all auth logs to the path /var/log/auth.log.

auth.* /var/log/auth.log

Specify a remote host: This forwards syslog messages to another host for further processing. The remote host’s syslog configuration will determine any further action once received.

• The syntax is protocol[zNUMBER]HOST:[PORT]
• The protocol is specified with @ for UDP or @@ for TCP
• The zNUMBER refers to the level of zlib compression (1-9)

The following example would forward all emergency-priority cron logs to 192.143.5.12 on port 18, using TCP and without compression.

cron.emergency @@192.143.5.12:18

Specify a user: This action specifies a username. By doing so, the user will see the message displayed on the console if they’re logged on. You can also use * to display the message for all users currently logged on.

For example, the following command would send all emergency-priority messages to user jbloggs123.

*.emergency jbloggs123

Execute a program: This allows you to execute a program on receipt of log messages. For example, an emergency-priority message could trigger a script to email the incident response team. The ^ symbol prefixes the program to execute, and you can create a template so the message is passed on in the correct format to the executable.

The example below processes all kernel messages and passes them on in executing a custom script.

kern.* ^/usr/local/bin/process-kernel-messages.sh

Discard a message: Adding a tilde symbol ~ will permanently delete matched messages.

This example will discard all user messages.

user.* ~

Write to a database: rsyslog natively supports MySQL and PostgreSQL, so you can write directly to tables within those databases. You may need to install the software packages rsyslog-mysql and rsyslog-pgsql. The following example shows the syntax for writing to a database.

:PLUGIN:DB_HOST,DB_NAME,DB_USER,DB_PASSWORD;[TEMPLATE]

Managing Log Files With Logrotate

Logrotate is a utility used to help manage log file sizes on the syslog server. It helps control the amount of storage consumed on the server to prevent it from running out of space unexpectedly.

Logrotate works by renaming the current log file when a certain condition is met—for example, if the file reaches a certain size or age. The rotated-out file is renamed for archival purposes, while new logs are then written to an empty file with the original filename.

Additionally, you can configure logrotate to compress older files, archive them to a new location, or even delete them altogether.

Logrotate configuration is defined in the /etc/logrotate.conf file. The configurable parameters include:

• daily: when to attempt to rotate the log file. You can choose between daily, weekly, monthly, or yearly to define the log rotation schedule.
• rotate: how many rotated log files should be retained. You can specify any integer (for example, rotate 6). Logrotate removes the oldest file when the next log file is rotated.
• dateext: whether to append the date to the log file name.
• size: trigger log rotation when the log file reaches a particular size limit (for example, size 10m).
• compress/delaycompress: whether to compress the rotated files to reduce their size.
• postrotate: execute a script after log rotation has taken place. For example, this could be to send an alert or to write to another log file.
• maxage: the maximum number of days to retain rotated log files before deleting (for example, maxage 60).

[user@host123@domain.com ~]$ cat /etc/logrotate.conf
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# compress rotated log files
compress
# rotate after 10m
size 10m
# run script
postrotate
/usr/local/bin/myscript
endscript
# delete any files over 32 days old
maxage 32

Syslog Best Practices

The following are best practices you should be aware of when working with syslog.

Plan your architecture design carefully

Syslog servers usually require a lot of storage and processing power, so it’s important to ensure the hardware is fit for purpose. Ensure you have enough storage, and consider a cloud environment for easier scaling. In globally distributed environments, configure clients to send to the closest syslog server to avoid latency issues.

Secure your logs

Ensure you use proper authentication protocols on your syslog servers so only authorized users have access. If you haven’t secured the syslog server properly, threat actors could access the logs from all of your servers. Remember to use TLS/SSL when forwarding sensitive logs, as this will ensure logs are safe in transit and are useless if intercepted.

Implement smart filtering policies

Logging everything can be expensive. Not only do you need storage space to hold the log messages, but you also need sufficient network bandwidth to transmit them all. Assess your logs and decide which logs to keep and which to discard. Maybe you could discard debug or informational logs or only retain logs from production hosts for a set period of time.

Retaining all logs can also make it difficult to find what you need. Search functions like grep may take longer to run if you have retained too many logs, and it may become increasingly difficult to separate important logs from noise.

Create backups regularly

If you’re subject to audits or have a regulatory obligation to retain data, then ensure you back up your syslog servers regularly. Additionally, backups help mitigate the risk of losing the syslog server.

Configure appropriate retention policies

Take advantage of logrotate on Linux machines to configure policies that rotate, archive, and delete data at appropriate time periods. For example, if you have 10GB of disk space, you could configure size 2G and rotate 3 to ensure logrotate clears disk space before reaching capacity. If no retention policies are in place, or if they are poorly configured, then syslog servers can quickly exceed capacity and become unusable.

Choose a communication protocol that suits your needs

Most people choose between UDP or TCP for transmitting data. If speed is more important to your organization, and you can deal with some data loss, then you should choose UDP. If data loss is not an option, and you can tolerate latency, then TCP may be more appropriate.

Log your data with CrowdStrike Falcon Next-Gen SIEM

Elevate your cybersecurity with the CrowdStrike Falcon^® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.

Schedule Falcon Next-Gen SIEM Demo