Concepts
Data Model
Understand the concepts behind the Streamfold data model and gain control over your Observability data.
Overview
Streamfold ingests data from Sources. Data received from sources are converted into Events. Streamfold events provide a common data format which makes it easy to work with data from any source. Events are processed by Streams and Functions before being delivered to their Destinations.
The following overview should help you better understand the Streamfold data model and gain control over your Observability data.
Events
The following is an abbreviated example of a log message event received from the Datadog Agent source. It is presented here in JSON, a format that roughly models the internal event model of Streamfold. Like JSON, an event is a collection of fields, or name-value pairs.
"_meta": {
"type": "log",
"source": {
"type": "datadog_agent"
},
"http": {
"method": "POST",
...
},
"peer": {
"client_ip": "172.17.0.2:54658"
}
},
"message": "2023-04-04 21:47:34 - hipstershop.AdService - received ...",
"status": "info",
...
}
All events are composed of two high level components: event fields and event meta data.
Event fields
Event fields are the actual contents of the event when it was received and parsed from the source. In the example above, message
and status
are fields of the individual log line received from the Datadog Agent. These will often vary from event to event. Many of the stream functions allow you to add or remove event fields.
Event fields are selected using their direct field names, like message
.
Event meta data
The _meta
field in the example above represents the meta data associated with the event. The meta data map, and all fields contained within it, are generated by the source at event ingestion time. These fields include identifiers of the source, the request parameters, and the type of the event. The meta data makes it easy to select events of the same type, in this case log events, from sources that support multiple event types.
We have represented the meta data section here with _meta
. However, as mentioned in the selector syntax, the fields of meta are selected using the @
prefix, so @type
references the value log.
Fields
As mentioned above, events are composed of multiple fields. All fields have a field name and a field value.
Field Names
Field names are case-sensitive strings and are unique with any given field map.
Field Values
Field values are value types and may be any of the following primitive types:
bool
integer
double
string
… or either of the following collection types:
map
array
.. or the special type
nil
Field Examples
The following are examples of fields. As previously mentioned fields are name-value pairs.
// name: "foo", value: "bar"
"foo" : "bar"
// name: "values", value: [0.09, 1.95, 200.0 ]
"values" : [0.09, 1.95, 200.0 ]
// name: "enabled", value: true
"enabled" : true
// name: "points", value: [ {"timestamp": 1685720550, "value" : 0.0 }]
"points" : [ {"timestamp": 1685720550, "value" : 0.0 }]
Field Nesting
As previously noted values may be of primitive types or a collection type map
or array
. Therefore, fields may be nested to an arbitrary depth.
Maps
Maps may only contain fields, that is primitive values cannot be stored as keys in a map
. Map keys are field names.
In the following example we have a field with a name of “peer”
with a value of type map
. The map
itself contains a field, whose name is “client_ip”
and value is of type string
, with the value of 127.0.0.1:57964
. The "peer"
field itself is nested in an outer map.
{
"peer": {
"client_ip": "127.0.0.1:57964"
},
}
Arrays
Arrays may contain primitive types, arrays, or maps.
In the next example we have a field with the name of "series"
and a value of type array
. Inside the series
array are a collection of maps. The maps each contain fields of various types. In some cases, the values are simple primitives like the field named metric
with a value "system.swap.free"
.
In some other cases fields are of the type array
. For example, the field named "tags"
has a value of type array
and it contains a collection of strings
. The field named "points"
is also of type array
and it contains a collection of maps, which further contain fields named "timestamp"
and "value"
.
{
"series": [
{
"tags": [
"version:7.42.1",
],
"type": 2,
"unit": "",
"interval": 10,
"metric": "system.swap.free",
"points": [
{
"timestamp": 1685720550,
"value": 0.000000
},
{
"timestamp": 1685720551,
...
},
...
],
...
},
{
"unit": "",
"interval": 10,
"metric": "datadog.trace_agent.trace_writer.retries",
"points": [
...
}
],
...
}
Accessing Field Values
In order to access the values of fields we use Selectors. Streamfold selectors are free-text path expressions which allow you to select values from fields within events. Selector path expressions uses dot notation, similar to JSON, to specify the full path to access a field's value.
As an example, in order to access the value for the field with the name of Content-Type
, we use the following path expression: http.headers.Content-Type
{
"http": {
"path": "/api/v2/logs",
"method": "POST",
"headers": {
"Content-Type": "application/json"
}
}
}
You can learn more about Streamfold selectors and path expression syntax under Reference - Selector Syntax