Humio Sets the Standard for Data Ingestion with Scalability Benchmark Streaming over One Petabyte of Data per Day

One petabyte, 30M events per second, all on only 45 nodes with 96 cores each

May 10, 2022

| Cinthia Portugal | Next-Gen SIEM & Log Management

This blog was originally published March 8, 2022 on humio.com. Humio is a CrowdStrike Company.

Humio is excited to achieve another milestone in data ingestion by reaching a

benchmark of over one petabyte of data ingestion per day. The Humio engineering team completed a one petabyte benchmark on only 45 nodes with 96 cores each, running 30 million events per second and delivering search results with sub-second latency through Humio’s index-free architecture. Other vendors have shown the need for 1200 nodes to reach only 100TB. This highlights Humio’s industry-leading total cost of ownership and ability to remove the limitations present in logging solutions, bringing unlimited ingest, reduced infrastructure costs, and lower operational costs. By using the same basic architecture in existing

Humio Quick Start configurations, enterprises can use the Humio platform to log everything to get answers to everything. The platform reaches 1PB/day at a fraction of the cost of traditional solutions due to unlimited ingest of log data, reduced infrastructure needs and lower operational costs. At the same time, Humio brings the benefits of a fast, flexible and powerful log management platform to aggregate, manage and use log data to make decisions across both the IT and business landscapes. “Log management is an important but often overlooked and underfunded side of IT. While data is growing, budget pressures often cause IT groups to scale back due to increasing infrastructure and operational costs faced when trying to achieve more observability and avoid blindspots,” said Steve McMahon, CIO and VP of Operations at CrowdStrike “We are excited to demonstrate the power Humio delivers to customers’ growing data ingestion needs, with 100% data fidelity from source to target, at an industry-leading TCO.” Humio enables complete

observability

to answer any question, explore threats and vulnerabilities, and gain valuable insights from all logs in real time. With this combination of massive scale and sub-second search capabilities, Humio provides a faster, more cost effective way to quickly and effectively deliver insights with a platform designed for ease of operations and stability at any scale. Several factors contribute to Humio’s ability to deliver such performance at this cost, including:

Architecture built for streaming.

The platform is optimized for the demands of modern log management and uses technologies like Apache Kafka to support streaming data continuous concurrent write and high-volume ingest of data in real time.
Index-free searches.

While legacy log management platforms index their data, this leads to search bottlenecks and the requirement to use significant amounts of disk space. Through index-free logging, Humio bypasses these issues to ingest and search logs faster. Now, enterprises can search 1 PB/day using Humio without the need to define and maintain complex indexes.
Data compression.

Humio uses advanced data compression algorithms that reduce the size of raw data by 10-20x. This drastically limits the amount of storage needed to retain log data on a massive scale.
Bucket storage.

Humio lets you write ingested logs to a native file format stored in Amazon Web Services S3 or Google Cloud Storage. This technique, called

bucket storage ,

markedly increases how much data can be stored in comparison to local disks. As a result, Humio cloud deployments are cheaper, faster, more scalable, easier to run (with ephemeral disks for primary storage), and more resilient against data loss.
Low licensing costs and overall TCO.

While other log management platforms penalize additional ingest, Humio’s licensing structure encourages it. Humio offers low licensing costs, including an

unlimited ingest

plan that allows enterprises to “log everything” with predictable pricing, resulting in a lower overall TCO.