Linux is an open-source operating system originating from the Unix kernel. It’s now one of the most used operating systems across devices. You may be familiar with the various flavors of Linux, including Ubuntu, Centos, and Red Hat Enterprise Linux (RHEL). Their common Linux core means all these operating systems have a logging framework installed to monitor the system and its services.

The logging framework for Linux includes a set of directories, files, services, and commands that administrators can use. As a Linux system administrator, knowing your way around the Linux log locations, commands, and configuration will be essential for troubleshooting issues on the systems or applications you administer.

This article is part one of a series on Linux logging. In part one, we will go through the basics of Linux logs: the common Linux logging framework, the locations of these log files, and the different types of logging daemons and protocols (such as syslog and rsyslog). Finally, we’ll review some common Linux log commands to read and search through the logs on a system.

Common Linux Logs and Their Locations

With the Linux logs pattern, you will find logs located under the /var/log directory, with files and directories for each service or stream of log messages on the system. Common log files include:

• /var/log/syslog (Debian) or /var/log/messages (RHEL): This is a consolidated stream of general system messages and metrics. In this log file, you can find messages from services like mail, kernel, authentication, and cron.
• /var/log/auth.log (Debian) or /var/log/secure (RHEL): This file contains authentication logs for both successful and failed login attempts on the system.
• /var/log/wtmp: This file contains a history of all user login and logout activity for auditing users' activity on the system.
• /var/log/lastlog: Similar to the wtmp audit file, this log file tracks users' last logins. This is a binary file you can read via the lastlog command.
• /var/log/kern.log: This file contains log messages produced by the kernel before being passed to the system logging service (such as rsyslog) for further processing.
• /var/log/cron: The cron service runs as an orchestrator to start tasks at scheduled intervals. Messages from this service—such as when a cron job has started and if any errors occurred during its execution—can be found in this log file.

When running services on your systems such as PostgreSQL or Apache, your application-specific logs are made available under subdirectories in /var/log. For example, if you are running an Apache web server on your Debian-based Linux system, you will find log files under the /var/log/apache2 directory. You would find separate files for each log stream in this directory, such as the access.log or the error.log.

Introduction to Syslog

Syslog is a network-based logging protocol that monitors your systems and applications. This protocol provides a standard way for services and applications to report their logs. That way, they can be processed and redirected as needed.

Standardized message format

The syslog protocol provides a message format defined by the RFC 5424 standard. In this format, common event information is defined, such as the timestamp, hostname, and the name of the application that produced the message. To further support the structuring of this message, syslog facilities are available to denote which part of the system the log comes from. This is done by attaching a number to the message. Below is a list of all available facilities, numbered from 0 to 23:

Similarly, priority can be attached to a message using a number between 0 and 7.

By using both the facilities and priorities in the syslog message, tools that access the syslog data can now filter messages based on the originating facility and the severity of messages. We’ll see an example of this in the next section.

Syslog Protocol Implementations

The syslog process runs as a daemon on the system to receive, store, and interpret syslog messages from other services or applications. That service typically listens on port 514 for TCP and 601 for UDP connections. Many applications allow you to configure their event logging to push messages to a running syslog service.

The syslog protocol is also implemented by different services like rsyslog and syslog-ng, allowing you to choose a service based on the feature set you need. Because these services have aligned to the syslog protocol, they are interchangeable for system and application logging, making them very scalable.

The Rsyslog Daemon

Rsyslog is a modern, open-source implementation of the syslog daemon, offering a high-performance, security-focused, modular design for any environment. The rsyslog daemon runs as a service on your host, listening for log messages sent to it and routing those messages based on defined actions.

In a typical installation of rsyslog, the daemon is configured through a file located at /etc/rsyslog.conf. In this config file, using selectors for the facilities and priority of the log message allows you to define what action should be carried out for the message. In the following example, any messages with the facility of mail and a priority of notice or higher will be written to a log file located at /var/log/mail_errors.

# <facility>.<severity>	<action>
mail.notice /var/log/mail_errors

These selectors are structured by facility (origin of the message) and priority (severity of the message), separated by a dot. The example below shows some possibilities of using this simple configuration to perform actions on incoming logs.

# Log a message to file
mail.notice /var/log/mail_errors
# Log a message to a user
Kern.debug bob
# Emergency messages from any facility should go to all users
*.emerg *
# Log a message to another host over UDP
*.* @remote-host
# Log a message to another host over TCP
*.* @@remote-host:514

Basic Commands for Linux Logging

As an administrator of Linux servers, you will often connect to these servers to read log messages for troubleshooting systems or the services running on them. Several utility commands are available on Linux systems, simplifying how you navigate stored log messages. The following section outlines some basic log commands available:

• cat: Short for concatenate, which allows you to view the contents of one or more files in the terminal.
• more: Similar to cat utility, this command reads the content of files in the terminal. However, this utility will interactively display it one page at a time to the user for an easier manual reading experience.
• less: Much like the more utility, this command displays a single terminal screen of content at a time, allowing for easier navigation of large text files.
• tail: By default, tail displays the last ten lines written to a file. Using the follow option (-f or --follow) allows you to monitor the file continuously. As new lines are written, they are printed to the user's terminal.
• head: This utility is the opposite of the tail command, fetching the beginning lines of a file. By default, head will display the first ten lines of a file.
• grep: This command allows you to parse input text using filters and regex to find specific patterns in the text. It is useful for searching and manipulating text in scripts or automation.

With these basic commands, you can easily access and navigate the log messages on your system. By using pipes (|) in your commands, you can chain multiple commands together, filtering their outputs even further. For example, the following chain of commands will read the contents of the /var/log/cron file and check if any message contains the string foo.

cat /var/log/cron | grep "foo"

Advanced logging operations can also be done with other commands like awk, cut, and advanced grep filters, allowing you to gain more insight into what happens on your system.

Log your data with CrowdStrike Falcon Next-Gen SIEM

Elevate your cybersecurity with the CrowdStrike Falcon^® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.

Schedule Falcon Next-Gen SIEM Demo