Selector Syntax

Introduction

Streamfold selectors are free-text path expressions which allow you to select values from events. If you're familiar with similar path expression languages like jq or have experience working with JSON, you'll find working with Streamfold selectors is a breeze.

Selector Grammar

property       selects property from an event’s Fields Map
@property      selects property from an event’s Meta Map
.property      selects property from a parent Map
foo.*          selects all properties from "foo" object
*.bar          selects all properties ending with "bar" in an object
[n]            select nth value from an Array
[start:end]    selects start inclusive, end exclusive values from an Array
[start:]       selects start inclusive to end of an Array
[:end]         selects from 0 to end exclusive values from an Array
[:]            selects all values from an Array
["foo.bar"]    selects value with key "foo.bar" from a Map

Selectors and the Streamfold Data Model

As noted in the Data Model there are two top level field types in Streamfold events.

• Meta fields - A collection of read only fields associated with an event's source, accessible in selectors using the @ prefix. Stored in a map at the root level of the event called _meta.
• Fields - A collection of mutable fields, received from the payload of the source, accessible with the . prefix. Stored at the root level of the event.

For example in the following event the selector .message and message will both successfully select the field named message from the root level of the event.

{
  "_meta": {
    "type": "log",
    "source": {
      "type": "datadog_agent"
    },
    "http": {
      "method": "POST",
      ...
    },
    "peer": {
      "client_ip": "172.17.0.2:54658"
    }
  },
  "message": "2023-04-04 21:47:34 - hipstershop.AdService - received ad request...",
  "status": "info",
  ...
}

Selecting Nested Fields

Streamfold events consist of fields, which are key/value pairs of names and values. The values may be of primitive types such as string, integer, double, or boolean, or may be a container type of either map or array. Nested values may be accessed in maps by using the . operator. For example to select the value for id in the following event fragment, we would select customer_info.account.id

{
  "customer_info": {
    "account": {
      "id": "<customer_id>"
    }
  }
}

You can also access values in a map using a specific key. Consider the following event fragment:

 "resourceLogs": [
  {
   "resource": {
    "attributes": [
     {
      "key": "telemetry.sdk.language",
      "value": {
       "stringValue": "python"
      }
     },
     {
      "value": {
       "stringValue": "opentelemetry-demo"
      },
      "key": "service.namespace"
     },
     ...

In this example, to access the value of a key named service.namespace in the resource.attributes map, you can use {resource.attributes["service.namespace"]}.

Selecting from Arrays

Streamfold events may also contain arrays. Values within arrays may be selected using array subscript syntax []. Individual indexes or ranges may be selected. Consider the following event fragment:

{
  "measurements" : [
    0.95, 0.87, 0.70
  ]
}

Selecting with wildcards

Streamfold's wildcard support, denoted by the asterisk (*), empowers users to easily select and filter data patterns within events, providing flexibility and efficiency in working with event data structures. For example, to select all of the properties within message.request in the following event fragment, including method and path, we could use the selector message.request.*.

{
  "hostname": "localhost",
  "message": {
    "level": "info",
    "ts": 1692835200.8646913,
    "caller": "server/request.go:126",
    "msg": "Finished request",
    "version": "0.0.1-a8ac28e",
    "request": {
      "method": "POST",
      "path" : "/api/v1/worker/ZH4C9WNFAVQ294AKBFNEC4FQ9M/heartbeat"
      ...
 }
    ...
  }

Using Selectors

Selectors are used in Stream Filters, Function Filters, and Functions. When selectors are used in Stream and Function Filters they're composed within Streamfold Expressions and evaluated for truthiness. Selectors are also used in functions to select which fields in an event to operate on.

Selectors in Stream and Function Filters

Consider the following event fragment received from a Datadog Source.

{
  "_meta": {
    "source": {
      "type": "datadog_agent",
      "id": "01GYZ5Y05D35E5ZN5YVBFV21MN",
      "name": "datadog_agent"
    },
    "http": {
      "path": "/api/v1/check_run",
      "method": "POST",
      "headers": {
        "User-Agent": [
       ... 
      }
    },
    "peer": {
      "client_ip": "127.0.0.1:57964"
    },
    "type": "check_run"
  },
  ... 
}

If we wish to construct a stream or function filter to operate on check_run type events only, we can use a selector to inspect the event's Meta fields and construct an expression to select the event's @type and see if it is equal to check_run.

If we wish to extend the filter to only pass on v1 style check runs we can add another condition to match on @http.path and ensure we set include events that match All filters.

Selectors in Functions

Selector path expressions are also used in functions to specify what field or fields a function should operate on. Consider the following event fragment. The event is a log message with @type == log and it was generated from an Apache server and ingested into Streamfold from a Datadog source. The message field contains the log line which is formatted using the Common Log Format.

{
  "_meta": {
    "type": "log",
    "http": {
      "method": "POST",
      ...
    }
  },
  "message": "1.1.1.2 access.log - [11/June/2023:03:04:55 +0100] " GET / " 200 83",
  ...
}

We can use the Extract function to parse the message field and extract new fields into the event for further processing.

{
  "_meta": {
    "type": "log",
    "http": {
      "method": "POST",
      ...
    }
  },
  "ip": "1.1.1.2",
  "remote_log_name": "access.log",
  "user_id": "-",
  "date": "[11/June/2023:03:04:55 +0100]",
  "request_method": "GET",
  "path": "/",
  "message": "1.1.1.2 access.log - [11/June/2023:03:04:55 +0100]  "GET /"  200 83",
  ...
}