Connect Fluent Bit to Streamfold

Fluent Bit is highly performant and lightweight log shipper. It can forward logs to a number of locations and supports basic log processing. By connecting Fluent Bit to Streamfold you unlock the ability to transform and control where your data is delivered to in real time with our UI and API.

This is a quickstart guide to getting started with Fluent Bit and Streamfold. In this guide you'll connect Fluent Bit to Streamfold and parse JSON structured logs with Streamfold.

On the completion of this guide you will have:

1. Connect your Fluent Bit agent to Streamfold
2. Send AWS ECS and Fargate logs to Streamfold
3. Parse JSON structured logs with Streamfold

Create an HTTP source

Fluent Bit supports an HTTP output plugin that we will use to send data to Streamfold. In Streamfold, navigate to the Source > Add Source page and select the HTTP source. Pick a name for your HTTP source and record the password presented in the authentication instructions. You'll need the password in later steps. Click Save to add the source.

Connect Fluent Bit

Fluent Bit can be configured in both Classic and YAML modes. We will demonstrate with the classic configuration.

Open your Fluent Bit configuration file, typically /etc/fluent-bit/fluent-bit.conf, in your editor and add the following section:

[OUTPUT]
    Name http
    Match *
    host ingress.streamfold.com
    port 443
    uri /streamfold/http
    http_User streamfold
    http_Passwd ${PASSWORD}
    TLS on
    compress gzip
    format json_lines
    json_date_key _timestamp
    json_date_format epoch
    header Content-Type application/x-ndjson
    header X-Service-Name ${SERVICE_NAME}

Replace the ${PASSWORD} field with the password you saved from the HTTP source. You can add optional header data to the integration, for example we are setting X-Service-Name to ${SERVICE_NAME}. This can be useful when filtering events in your Streamfold stream.

This output plugin configuration will match all Fluent Bit records and forward them to Streamfold. Check out the full configuration documentation for more instructions.

Connect AWS Firelens

AWS Firelens is a new log router for Amazon ECS and AWS Fargate that leverages Fluent Bit. Firelens allows you to stream logs out of your ECS and Fargate containers without having to rely on AWS Cloudwatch. You can use Firelens to stream your logs directly to Streamfold, allowing you to transform, filter or route them to any number of supported destinations.

Familiarize yourself with the AWS guide for using Firelens with your application. You will need to add the aws-for-fluent-bit container to your ECS task definition. Once you've added that, update the logsConfiguration for your application's container to the following:

"logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
                "Name": "http",
                "Host": "ingress.streamfold.com",
                "URI": "/streamfold/http",
                "http_user": "streamfold",
                "http_passwd": "${PASSWORD}",
                "Port": "443",
                "TLS": "on",
                "Compress": "gzip",
                "Format": "json_lines",
                "Json_date_key": "_timestamp",
                "Json_date_format": "epoch",
                "header": "Content-Type application/x-ndjson",
                "header": "X-Service-Name ${SERVICE_NAME}",
        }
}

Replace ${PASSWORD} to match the password from your HTTP source. You can add optional headers like X-Service-Name to your configuration as well.

You can use secretOptions to set options from secrets stored in AWS Secrets Manager instead of plain text, like:

"secretOptions": [{
        "name": "http_password",
        "valueFrom": "arn:${Partition}:secretsmanager:${Region}:${Account}:secret:${SecretId}"
}]

Connect to a stream

Follow the guidance from the Getting Started Guide to connect your Streamfold HTTP source to a destination and stream. You can also use the event sampler to sample the log events entering your HTTP source.

Parsing JSON-structured logs

If your application emits structured JSON logs, you may find that you need to parse the inner JSON encoded string fields. Some destinations will handle this by default, like Datadog, but others like S3 may require a transformation.

Assume you have a log message that is emitted by Fluent Bit like:

{
  "source": "stderr",
  "ecs_cluster": "api",
  "container_name": "streamfold-api",
  "message": "{\"hello\": \"world\"}"
}

In your Streamfold stream, click to add a function and select Parse. For the Field Name enter message and select JSON as the format. For a destination, pick an unused field like msg. Enable overwrite destination if data exists and click Save.

When this function executes on a event you'll see that a new field is added to the top-level of your event. Subsequent functions can operate on this data as if it were in the original event.

"msg": {
  "hello": "world"
}

Next steps

Now that you've connected your Fluent Bit or AWS Firelens traffic to Streamfold you are on your way to unlocking your telemetry.

• Setup multiple destinations to archive log data while maintaining operational visibility of the important log streams.
• Drop verbose logs to reduce your operational spend.
• Grok fields from logs lines to enrich the field data in your logs.