HTTP Event Collector (HEC)

Easily ingest data to Streamfold with Splunk's HTTP Event Collector (HEC) API.

The HTTP Event Collector (HEC) API allows you to ingest large amounts of event data efficiently. Payloads can be sent in JSON format or as lines of raw text. This API is often used by Splunk log shippers and other community projects.

Overview

The HEC source can accept multiple events sent in either JSON format or as raw text, separated by newlines. Each submission will be converted into multiple events in the Streamfold telemetry pipeline. The individual formats are described below with their respective endpoints.

Submit JSON

Either of the following endpoints will accept HEC formatted JSON:

• POST /streamfold/hec
• POST /services/collector

This is an example of two JSON-encoded events sent to the endpoint:

{
  "time" : 1705377163.0,
  "event" : "Tue Jan 16 03:52:43 2024 - system started server=puma",
  "source" : "/var/log/webapp.log",
  "fields" : { "env" : "prod" }
}

{
  "time" : 1705377231.01234,
  "event" : {
    "message" : "System shutdown",
    "host" : { "ip" : "10.0.0.1", "port" : 3000 }
  },
  "fields" : { "datacenter" : "us-east-1" }
}

If the time field is present in a JSON event, it is interpreted as the timestamp of the event. The value is parsed as epoch seconds with millisecond accuracy.

RAW

You can send raw text lines separated by a newline to the raw endpoint. Both of the following endpoints may be used to send raw text:

• POST /streamfold/hec/raw
• POST /services/collector/raw

This is an example of two text events sent to the raw endpoint:

2024-01-15 20:05:49,620 - util.py[DEBUG]: Read 13 bytes from /proc/uptime
2024-01-15 20:05:49,620 - util.py[DEBUG]: cloud-init mode 'modules' took 76.982 seconds (75.78)

When they are converted to an event, each line of text is set to the top-level key event as such:

{
  event: "2024-01-15 20:05:49,620 - util.py[DEBUG]: Read 13 bytes from /proc/uptime"
}
{
  event: "2024-01-15 20:05:49,620 - util.py[DEBUG]: cloud-init mode 'modules' took 76.982 seconds (75.78)"
}

Authentication

See the Source Authentication page for how to authenticate your log shipper with Streamfold.

Content encodings

You can optionally encode request bodies with the following content encoding types.

Content-Encoding: gzip

Request body will be decompressed with gzip.

Content-Encoding: deflate

Request body will be decompressed with zlib.