Falcon LogScale S3 ingest

November 9, 2023

| |Tech Hub•Next-Gen SIEM

CrowdStrike Falcon LogScale now has the ability to ingest logs from AWS S3 buckets, in this blog we will be running through the configuration process of ingesting this data.

 

Amazon Web Services log data is an extremely valuable data source that comes in a variety of flavors depending on the services you are looking to learn more about. Some of the most common data sources being AWS VPC flow, Cloudtrail and CloudWatch. These logs can be directed to S3 buckets where they are often ingested by log management or SIEM solutions. More details on these logs can be found here. 

 

Prerequisites

To follow the steps, you will need access to AWS, basic knowledge of AWS architecture and access to a Falcon LogScale environment as well as an understanding of its architecture.

 

This article assumes you have full permissions and access to your S3 buckets. If not, you will need to configure permissions to allow S3 integration Falcon LogScale.

 

Writing Logs to S3

Your first step is to make sure that your AWS services are writing their logs to S3. This can be to a separate bucket or a directory within a bucket. These log files will then be pulled into Falcon LogScale for analysis and visualization, the format of the data can be line-delimited or AWS JSON events. AWS example events are referenced here.

 

Setting up a Repository in Falcon LogScale

The first step in the process is to configure a repository so that we can have a target for the logs we are ingesting and that they can be stored, searched and analyzed using Falcon LogScale.

 

Configure the Simple Queue Service 

A new SQS will need to be created and will notify us when new logs are available in the S3 bucket for ingest. The configuration of the SQS will require properties such as a name, retention period in days before they are deleted:

AWS

We will then need to setup an access policy under advanced settings:

Access Policy

This is where we retrieve the Amazon Resource Name and specify it in the configuration to give it access to the S3 bucket and allow it to send messages as follows:

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:*",
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "S3_BUCKET_ARN"
                }
            }
        }
    ]
}

 

Then choose the option to create a queue.

 

The next step is to configure the S3 bucket to create event notifications: 

Configure the SE bucket

Ensure you select all object create events:

Select all object create events

Ensure you select the SQS you created in the earlier step:

Select the SQS

You should at this stage go to the SQS queue and check the monitoring tab to see the Number of messages being sent. This process may take a few minutes:

SQS queue

Configure IAM Role

The next step is to setup a role in IAM to allow reading of the data:

Setup IAM Role

Select Roles and Create a Role with a custom trust policy:

Create Role

Use the information retrieved from the Falcon LogScale configuration to populate the custom trust policy. To retrieve this information, go to Falcon LogScale > Settings > Ingest feeds > Add AWS ingest feed.

Add AWS ingest feed

You have a variety of options to select including compression options such as Auto, Gzip or none. There are also preprocessing options to Split by AWS records or Split by new line. You then also have the ability to select the parser depending on the format or structure of your data.

 

When you select next you will be able to retrieve the IAM identity and External ID details so they can be specified in the custom trust policy:

Add credentials

The Trust policy should then look like:

{

    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLogScaleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "LOGSCALE_AWS_IDENTITY_ARN"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "LOGSCALE_EXTERNAL_ID"
                }
            }
        }
    ]
}

You will then be able to provide a role name and then finish by choosing create role.

 

Once the role is created, you can search for it and select it:

Select Role

The next step is to create an inline policy

Create inline policy

The inline policy allows us to receive the notifications we configured and to read the logs from the S3 buckets, we will need information such as the S3 bucket ARN and the SQS ARN. The inline policy will be in the following format:

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReceiveS3Notifications",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "SQS_QUEUE_ARN"
        },
        {
            "Sid": "ReadS3Objects",
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "S3_BUCKET_ARN/*"
        }
    ]
}

 

Once you click next you will be asked to specify a policy name and complete the process by choosing the Create policy option.

 

For the final part of the process you will specify the role ARN, SQS URL and AWS region within the LogScale configuration:

Add Credentials

Ingest feeds will look like the following:

Ingest Feeds

You should now have data flowing into CrowdStrike Falcon LogScale from AWS S3 buckets.

 

Unbeatable Scale and Performance

Join leading organizations by augmenting or replacing your security information and event management (SIEM) solution with Falcon LogScale for unbeatable scale and performance. With its unique index-free architecture and advanced compression technology, Falcon LogScale empowers you to stop breaches by delivering high-speed search and sub-second latency for live dashboards and real-time alerting.

 

With Falcon LogScale, you can cut costs by up to 80% compared to legacy SIEM solutions. Its vast scale and affordable price let you avoid making tough tradeoffs between cost, how much data you can collect and how long you can store it. With Falcon LogScale, you can retain petabytes of data for years.

 

To learn more about Falcon LogScale integrations, visit the Integrations page. To find out if Falcon LogScale can help you fulfill your SIEM and logging requirements, contact a CrowdStrike expert today.

 Additional Resources

Related Content
NG-SIEM – AI Assisted Investigation
NG-SIEM – Adversary Driven Detection
NG-SIEM – Streamline The SOC