Streamfold - Functions

The Extract function reads data from a specified field and extracts the contents into new fields as specified by a Go or Grok expression containing capture groups.

Usage

Operates On: Field values
Supported Field Types: string
Label: Optional
Filter: Optional

Required Configuration

Field Name: The field to extract data from.
Expression Type: Go Regular Expression | Grok.
Expression: A Go regular expression or Grok expression containing capture groups.
Overwrite Existing Fields: True | False. When enabled will overwrite any existing fields whose field name is the same as extracted field name.

Optional Configuration

Destination Field: type: string - path selector. Destination field name to write extracted fields to. If unset the extracted fields will be set at the same level in the event structure as the field name.

Grok example

Extract fields from an event's message field, using a Grok expression. The Grok Constructor is a useful site for testing Grok patterns.

Configuration

Field Name: message
Expression Type: grok
Expression: %{IP:ip} \[%{HTTPDATE:timestamp}\] "%{WORD:request_method} %{DATA:path}" %{NUMBER:status}
Overwrite Existing Fields: true

Result

Input:

message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",

Output:

ip: "1.1.1.2"
path: "/"
timestamp: "11/June/2023:03:04:55·+0100"
status: "200"
request_method: "GET"
message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",

Go Regex example

This example uses Go regular expressions (regexp) to extract matching fields from an event field. The regex101 site is useful for testing Go regular expressions.

Configuration

Field Name: message
Expression Type: goregex
Expression: RequestId: (?P<request_id>[-0-9a-z]+)[ ]+Duration: (?P<duration>[.0-9]+) ms
Overwrite Existing Fields: true

Result

Input:

message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95  Duration: 15.74 ms"

Output:

message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95  Duration: 15.74 ms"
request_id: "8f507cfc-xmpl-4697-b07a-ac58fc914c95",
duration: "15.74"

Example Expressions

Below are some examples of how to construct regular expression with capture groups in Go and Grok. In these examples we've constructed expressions to parse CLF log messages. For more information on constructing expression view the Go regular expressions and Grok expression docs linked above.

Go regular expression to parse CLF

(?P<ip>.*?) (?P<remote_log_name>.*?) (?P<userid>.*?) \[(?P<date>.*?) (?P<timezone>.*?)\] \"(?P<request_method>.*?) (?P<path>.*?)(?P<request_version> HTTP/.*)?\" (?P<status>.*?) (?P<length>.*?) \"(?P<referrer>.*?)\" \"(?P<user_agent>.*?)\" (?P<session_id>.*?) (?P<generation_time_micro>.*?) (?P<virtual_host>.*)

Grok regular expression to parse CLF

%{IP:ip} %{DATA:remote_log_name} %{DATA:userid} \[%{HTTPDATE:date}\] \"%{WORD:request_method} %{DATA:path}\" %{NUMBER:status} %{NUMBER:length} \"%{DATA:referrer}\" \"%{DATA:user_agent}\" %{DATA:session_id} %{NUMBER:generation_time_micro} %{GREEDYDATA:virtual_host}