A More Modern Approach to Logging in Go

February 08, 2022

| | Engineering & Tech
The Go ecosystem has long relied on the use of third-party libraries for logging. Logrus, one of the first leveled, structured logging libraries, is now maintenance-only and its developers recommend migrating to other libraries.

 

 At CrowdStrike, we relied heavily on Logrus and recently underwent an overhaul to implement a more modern approach to logging. In evaluating our options, our team needed to find an alternative logging library that fit our criteria of being highly performant, having a low memory profile and supporting CrowdStrike’s unique formatting needs. In this post, we outline the options available in a post-Logrus world and explore our decision-making process to identify the best solution based on our unique needs.

 

Exploring the Go Libraries

Logging in Go has evolved over the years as engineers have pursued the ideal of a highly intuitive API with zero allocations and low performance overhead. While Logrus was a solid starting point, it is no longer a viable option. There are two main libraries to consider when choosing a replacement: Zap and Zerolog.

Zap

Zap is a leveled logging library that's written and maintained by Uber. Zap has an easy-to-use, intuitive API that allows for easy customization. While it was launched after Logrus, it was introduced before other currently favored, more performant libraries were available.

 

 The main attraction for Zap is that it’s easily extensible to support a wide range of formats, such as Logfmt. It also has numerous customization options such as multi-writers, overridable default fields, hooks and a familiar API thanks to wide adoption. That said, it tends to be slower than newer libraries, which can be a concern for organizations that process extreme amounts of data.

 

Pros:

 

  • SugaredLogger allows for interface typing without sacrificing strictly typed performance
  • Easily extensible thanks to abstractions
  • Supports JSON natively, as well as a wide range of other formats via encoder interfacing
  • Supports a wider range of functionality compared to alternatives

Cons:

  • Slower than some other implementations by virtue of having more levers and abstractions
  • No native Syslog support (see this Github issue)
  • Managing Cores and switching in and out of SugaredLogger adds complexity

Zerolog

Zerolog is a zero allocation JSON logger inspired by Zap. While Zerolog also offers the option of using build flags for encoding as Concise Binary Object Representation (CBOR), it isn’t extensible to other formats. In other words, if your ecosystem supports logging as JSON, it’s highly recommended to use Zerolog. While there’s some performance benefit to Zerolog’s approach of using build flags to set the encoder rather than abstracting encoding with an interface, this approach has some drawbacks. For example, Zerolog’s developer-friendly ConsoleWriter will decode JSON stored in the internal byte buffer, then re-encode the log message in K=V format. Decoding the byte buffer, then encoding to the desired output is a significant performance loss. Further, because the encoder is stored as a private package level variable, it’s not possible to override the default encoder to make ConsoleWriter more performant or provide your own encoding.

Pros:

  • Excellent choice for ecosystems that support JSON formatted logs
  • Faster than Zap with fewer allocations
  • Developer-friendly console writer
  • Simple, straightforward API

Cons:

 

  • Can only output log lines in two specific formats: JSON and the not widely used CBOR
  • Less flexible than Zap

Exploring the Interfaces

The pseudo-deprecation of Logrus highlights the need for organizations to keep their internal ecosystems decoupled from dependence on third-party libraries. If your organization is forced to migrate to a new library, now is an ideal time to consider preventing future migration headaches by adding a logging interface.

 

 

Two Interface Options: logr and DIY

The first option for avoiding vendor lock-in and standardizing your logging APIs is logr, a fairly opinionated library that defines an interface for structured, leveled logging. Benefits include:
  • Reduced API surface by removing named levels
  • Reinforcement of structured logging by removing string formatting (e.g., .Infof()
  • Variadic API that allows easier optimization as compared to relying on maps
This logr FAQ explores some of the specific benefits to abstracting your logger implementation in a well-thought-out interface. The other contender to consider at the interface level is to do it yourself (DIY). This is the route we used for CrowdStrike cloud services since our needs differed from what was available in the logr interface and the available library adapters. Because we were migrating several existing services from Logrus, we needed an API that was both easy to use and easy to migrate. We found that the delta between Logrus’s API and the logr API was a bit higher than we wanted for a fairly seamless transition. While rolling your own interface offers a lot of flexibility and customization, it has some notable drawbacks, the most important being that it will need to be self-maintained (as opposed to benefiting from updates from the community).

 

 Another important drawback to introducing an interface layer is the potential impact on performance. It is highly recommended to benchmark and test for data races early and often while developing your interface and adapters as there are many issues that may have a significant impact on performance and stability.

A Note on Formatting

A number of data formats useful for logging have been standardized. In some cases, they are logging specific (as with logfmt) while others have much broader usage. There’s no right or wrong answer for which to implement, so long as you standardize and build tooling around one format consistently. Options include:

JSON

  • Ubiquitous, supported by a wide variety of tooling
  • Log aggregator support via Humio and Splunk
  • Default encoder for most logging libraries

Logfmt

  • Heroku’s log format specification
  • Mixed support
  • Supported by Humio

     

  • Requires custom field definitions in Splunk

RFC5424/Syslog

  • RFC5424 defines log formatting for usage in Syslog

     

  • Library support for Syslog is a mixed bag — neither Zap nor Zerolog support Syslog formatting natively
  • Log aggregators like Humio and Splunk support ingestion of Syslog formatted logs

     

CBOR

  • CBOR provides a compact alternative to JSON where data size is a concern
  • Industry support is lacking compared to other formats

A Closer Look at CrowdStrike’s Journey

CrowdStrike processes trillions of events per week. At that scale, it’s absolutely critical that our common libraries are highly performant. We evaluated a number of public benchmarks when comparing the most popular logging libraries:
  1. Zap benchmarks
  2. Zerolog benchmarks
  3. Log Bench
Based on these benchmarks, which aligned with our internal testing, we adopted Zerolog as our logging library.

Looking Back: 2011 to Trillions

CrowdStrike started with a global singleton wrapping Logrus. This made it easy for us to standardize logging across all our Go libraries and services without having to do any dependency injection. While this gave us a ton of development velocity by simplifying dependency management, it also had its fair share of drawbacks.
  • One logger to rule them all. We couldn’t customize the logger per library/package. If an engineer wanted to limit error-level logging in the Kafka library, that also limited error-only logging in the service.
  • “Everything and the kitchen sink” API. With one logging API needing to account for every use case, the external API became bloated and was not user friendly.
  • High migration cost. Having built hundreds of services around a single library, CrowdStrike came to possess a large number of touch points that would all need to be updated if and when we decided to migrate. Moving away from Logrus wasn’t as simple as updating our internal adapter; we also had to update every touchpoint in every service that made use of Logrus’ specific API (which included heavy use of Fields).

     

In building out our new logging API, we leaned heavily on logr’s guiding principle of providing a simple and elegant structured logging interface. Between the performance gains of migrating from Logrus to Zerolog and the addition of our hand-rolled logging interface, CrowdStrike now has an easy-to-use, highly performant logging solution for years to come. In 2021, CrowdStrike saw that log management was a growing and critical part of IT, DevOps and SecOps, and purchased Humio, a leading provider of log management solutions. With Humio, organizations get a cost-effective and scalable way to aggregate log data from across their IT infrastructure.

What’s Next for Your Library?

With Logrus sunsetting, what’s your organization's migration plan? What advantages or drawbacks would you share with the Go community as the logging ecosystem evolves? Sound off on social media via @CrowdStrike.
Breaches Stop Here
Related Content
Tech Analysis: Channel File May Contain Null Bytes
Tech Analysis: Channel File May Contain Null Bytes
EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis
EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis
CrowdStrike Falcon Next-Gen SIEM Unveils Advanced Detection of Ransomware Targeting VMware ESXi Environments
CrowdStrike Falcon Next-Gen SIEM Unveils Advanced Detection of Ransomware Targeting VMware ESXi Environments