HTTP

Send events to any HTTP endpoint.

The HTTP destination connects Streamfold to any external destination that supports generic HTTP endpoints. This can be used to connect Streamfold to other cloud services.

Overview

The HTTP destination connects Streamfold to external services that support generic HTTP endpoints. Event data is encoded in JSON format and sent in batched payloads to the configured endpoint. This is ideal for streaming events to external data services.

Multiple events will be batched and sent in a single HTTP request. Events will encoded to JSON and batched in a single JSON array or in NDJSON format, with one event per line. The size, maximum events, and batch wait times for can all be configured.

Configuration

The HTTP destination supports a number of configuration options that control how HTTP are requests are made.

Common options

• URL: Full URL of external HTTP endpoint include schema, host and path. E.g. https://foo.example.com/request/in
• Format: Events are submitted as either json (as an array of JSON events) or ndjson (one JSON event per line, separated with newline). Default is json
• Compression: Whether to compress payloads, options are: gzip or zlib. Default is no compression
• Basic auth: Set basic auth username and password for outgoing requests

Advanced options

• HTTP Method: Set the HTTP method used to request the endpoint, default is POST
• Batch max bytes: Control the maximum byte size of HTTP body, default is 5MB
• Batch max events: Maximum number of events per batched HTTP request, default is 1,000
• Batch max age: Maximum time to wait when batching events before sending a batch, default is 30 secs
• HTTP timeout: Total maximum duration for an HTTP request to complete, default is 5 secs.

Notes

The HTTP destination will consider any 2xx response code a success. Response codes of 400, 413, and 403 will drop events. Any other response code greater than 400 or that results in an error/timeout will be retried.

The HTTP destination may maintain multiple concurrent connections to the HTTP endpoint. Connections will be reused if keep-alive is supported, otherwise a new connection will be made on each request.