Data Model

Understand the concepts behind the Streamfold data model and gain control over your Observability data.

Overview

Streamfold ingests data from Sources. Data received from sources are converted into Events. Streamfold events provide a common data format which makes it easy to work with data from any source. Events are processed by Streams and Functions before being delivered to their Destinations.

The following overview should help you better understand the Streamfold data model and gain control over your Observability data.

Events

The following is an abbreviated example of a log message event received from the Datadog Agent source. It is presented here in JSON, a format that roughly models the internal event model of Streamfold. Like JSON, an event is a collection of fields, or name-value pairs.

  "_meta": {
    "type": "log",
    "source": {
      "type": "datadog_agent"
    },
    "http": {
      "method": "POST",
      ...
    },
    "peer": {
      "client_ip": "172.17.0.2:54658"
    }
  },
  "message": "2023-04-04 21:47:34 - hipstershop.AdService - received ...",
  "status": "info",
  ...
}

All events are composed of two high level components: event fields and event meta data.

Event fields

Event fields are the actual contents of the event when it was received and parsed from the source. In the example above, message and status are fields of the individual log line received from the Datadog Agent. These will often vary from event to event. Many of the stream functions allow you to add or remove event fields.

Event fields are selected using their direct field names, like message.

Event meta data

The _meta field in the example above represents the meta data associated with the event. The meta data map, and all fields contained within it, are generated by the source at event ingestion time. These fields include identifiers of the source, the request parameters, and the type of the event. The meta data makes it easy to select events of the same type, in this case log events, from sources that support multiple event types.

We have represented the meta data section here with _meta. However, as mentioned in the selector syntax, the fields of meta are selected using the @ prefix, so @type references the value log.

Fields

As mentioned above, events are composed of multiple fields. All fields have a field name and a field value.

Field Names

Field names are case-sensitive strings and are unique with any given field map.

Field Values

Field values are value types and may be any of the following primitive types:

• bool
• integer
• double
• string

… or either of the following collection types:

• map
• array

.. or the special type

• nil

Field Examples

The following are examples of fields. As previously mentioned fields are name-value pairs.

// name: "foo", value: "bar" 
"foo" : "bar"  

// name: "values", value: [0.09, 1.95, 200.0 ]
"values" : [0.09, 1.95, 200.0 ]

// name: "enabled", value: true
"enabled" :  true
        
// name: "points", value: [ {"timestamp": 1685720550, "value" : 0.0 }] 
"points" : [ {"timestamp": 1685720550, "value" : 0.0 }]

Field Nesting

As previously noted values may be of primitive types or a collection type map or array. Therefore, fields may be nested to an arbitrary depth.

Maps

Maps may only contain fields, that is primitive values cannot be stored as keys in a map. Map keys are field names.

In the following example we have a field with a name of “peer” with a value of type map. The map itself contains a field, whose name is “client_ip” and value is of type string, with the value of 127.0.0.1:57964. The "peer" field itself is nested in an outer map.

{
  "peer": {
    "client_ip": "127.0.0.1:57964"
  },
}

Arrays

Arrays may contain primitive types, arrays, or maps.

In the next example we have a field with the name of "series" and a value of type array. Inside the series array are a collection of maps. The maps each contain fields of various types. In some cases, the values are simple primitives like the field named metric with a value "system.swap.free".

In some other cases fields are of the type array. For example, the field named "tags" has a value of type array and it contains a collection of strings. The field named "points" is also of type array and it contains a collection of maps, which further contain fields named "timestamp" and "value".

{
  "series": [
    {
      "tags": [
        "version:7.42.1",
      ],
      "type": 2,
      "unit": "",
      "interval": 10,
      "metric": "system.swap.free",
      "points": [
        {
          "timestamp": 1685720550,
          "value": 0.000000
        },
        {
          "timestamp": 1685720551,
          ... 
        },
        ...
      ],
      ...
    },
    {
      "unit": "",
      "interval": 10,
      "metric": "datadog.trace_agent.trace_writer.retries",
      "points": [
        ...
    }
  ],
  ...
}

Accessing Field Values

In order to access the values of fields we use Selectors. Streamfold selectors are free-text path expressions which allow you to select values from fields within events. Selector path expressions uses dot notation, similar to JSON, to specify the full path to access a field's value.

As an example, in order to access the value for the field with the name of Content-Type , we use the following path expression: http.headers.Content-Type

{
  "http": {
    "path": "/api/v2/logs",
    "method": "POST",
    "headers": {
      "Content-Type": "application/json"
    }
  }	
}

You can learn more about Streamfold selectors and path expression syntax under Reference - Selector Syntax