Support

Generic Functions

Extract

The Extract function reads data from a specified field and extracts the contents into new fields as specified by a Go or Grok expression containing capture groups.


Usage

  • Operates On: Field values
  • Supported Field Types: string
  • Label: Optional
  • Filter: Optional

Required Configuration

  • Field Name: The field to extract data from.
  • Expression Type: Go Regular Expression | Grok.
  • Expression: A Go regular expression or Grok expression containing capture groups.
  • Overwrite Existing Fields: True | False. When enabled will overwrite any existing fields whose field name is the same as extracted field name.

Optional Configuration

  • Destination Field: type: string - path selector. Destination field name to write extracted fields to. If unset the extracted fields will be set at the same level in the event structure as the field name.

Grok example

Extract fields from an event's message field, using a Grok expression. The Grok Constructor is a useful site for testing Grok patterns.

Configuration

  • Field Name: message
  • Expression Type: grok
  • Expression: %{IP:ip} \[%{HTTPDATE:timestamp}\] "%{WORD:request_method} %{DATA:path}" %{NUMBER:status}
  • Overwrite Existing Fields: true

Result

Input:

message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",

Output:

ip: "1.1.1.2"
path: "/"
timestamp: "11/June/2023:03:04:55·+0100"
status: "200"
request_method: "GET"
message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",

Go Regex example

This example uses Go regular expressions (regexp) to extract matching fields from an event field. The regex101 site is useful for testing Go regular expressions.

Configuration

  • Field Name: message
  • Expression Type: goregex
  • Expression: RequestId: (?P<request_id>[-0-9a-z]+)[ ]+Duration: (?P<duration>[.0-9]+) ms
  • Overwrite Existing Fields: true

Result

Input:

message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95  Duration: 15.74 ms"

Output:

message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95  Duration: 15.74 ms"
request_id: "8f507cfc-xmpl-4697-b07a-ac58fc914c95",
duration: "15.74"

Example Expressions

Below are some examples of how to construct regular expression with capture groups in Go and Grok. In these examples we've constructed expressions to parse CLF log messages. For more information on constructing expression view the Go regular expressions and Grok expression docs linked above.

Go regular expression to parse CLF

(?P<ip>.*?) (?P<remote_log_name>.*?) (?P<userid>.*?) \[(?P<date>.*?) (?P<timezone>.*?)\] \"(?P<request_method>.*?) (?P<path>.*?)(?P<request_version> HTTP/.*)?\" (?P<status>.*?) (?P<length>.*?) \"(?P<referrer>.*?)\" \"(?P<user_agent>.*?)\" (?P<session_id>.*?) (?P<generation_time_micro>.*?) (?P<virtual_host>.*)

Grok regular expression to parse CLF

%{IP:ip} %{DATA:remote_log_name} %{DATA:userid} \[%{HTTPDATE:date}\] \"%{WORD:request_method} %{DATA:path}\" %{NUMBER:status} %{NUMBER:length} \"%{DATA:referrer}\" \"%{DATA:user_agent}\" %{DATA:session_id} %{NUMBER:generation_time_micro} %{GREEDYDATA:virtual_host}
Previous
Drop