OpenTelemetry

Send metrics, logs, or traces to any OpenTelemetry (OTLP) compatible endpoint.

Overview

The OpenTelemetry destination connects Streamfold to external services that support the OpenTelemetry Protocol (OTLP). The destination supports OpenTelemetry metrics, logs, and traces. It can post to endpoints that support OTLP/gRPC or OLTP/HTTP in either JSON or Protobuf format.

When you combine it with the OpenTelemetry source your OpenTelemetry data is passed directly through to the destination. See below for how non OpenTelemetry events are auto converted.

Configuration

Endpoint options

You must select the OTLP wire format to use, the options are:

• OTLP/gRPC
• OTLP/HTTP - JSON
• OTLP/HTTP - Protobuf

The endpoint should be specified in URL format with optional port, like: https://api.example.com:4817.

For OTLP/HTTP endpoints you can override the path for metrics, logs, or traces. By default they will use /v1/metrics, /v1/logs, and /v1/traces, respectively.

By default the destination will compress payloads with gzip, but you can also disable compression.

Authentication and metadata

To support authentication or to provide additional metadata on requests, you can specify additional headers to inject on all requests. These headers are added as HTTP headers for OTLP/HTTP endpoints or as gRPC metadata fields for OTLP/gRPC endpoints.

OpenTelemetry conversion

If you connect a native OpenTelemetry data source to the OpenTelemetry or Honeycomb destinations to your stream, Streamfold will pass the OTLP formatted data along directly. This includes support for OpenTelemetry metrics, logs, and traces.

If an event is not already in OTLP format, Streamfold will attempt to convert it to OTLP before sending it to the destination. The conversion will attempt to convert events to OpenTelemetry logs by default. During conversion, it will include all top-level scalar event fields as log attributes. Attributes that match the stable semantic resource attribute conventions (service.name, service.version, telemetry.sdk*) will be hoisted to resource-level attributes during the conversion.

For example, for the following incoming event:

service.name: "acme-www"
url: "https://acme.example.com/store/8221.html"
method: "GET"
response_size: 2073
duration: 204.98

Streamfold will convert the event to an Open Telemetry log record with the following properties:

• Resource attribute: service.name of acme-www
• Log record attributes:
  • url (string): "https://acme.example.com/store/8221.html"
  • method (string): "GET"
  • response_size (integer): 2073
  • duration (float64): 204.98

You may find it useful to convert field names to their semantic convention prior to conversion by including a Set or Bloblang function in your stream.

Notes

• At the moment, map and array types are not supported when converting to OpenTelemetry. Fields of these types will be excluded from conversion.
• We plan to provide more options to control how events are converted to OpenTelemetry in the future.

If you have feedback on how to improve OpenTelemetry conversion, please let us know!