Generic Functions
Extract
The Extract function reads data from a specified field and extracts the contents into new fields as specified by a Go or Grok expression containing capture groups.
Usage
- Operates On: Field values
- Supported Field Types: string
- Label: Optional
- Filter: Optional
Required Configuration
- Field Name: The field to extract data from.
- Expression Type: Go Regular Expression | Grok.
- Expression: A Go regular expression or Grok expression containing capture groups.
- Overwrite Existing Fields: True | False. When enabled will overwrite any existing fields whose field name is the same as extracted field name.
Optional Configuration
- Destination Field: type: string - path selector. Destination field name to write extracted fields to. If unset the extracted fields will be set at the same level in the event structure as the field name.
Grok example
Extract fields from an event's message
field, using a Grok expression. The Grok Constructor is a useful site for testing Grok patterns.
Configuration
- Field Name:
message
- Expression Type:
grok
- Expression:
%{IP:ip} \[%{HTTPDATE:timestamp}\] "%{WORD:request_method} %{DATA:path}" %{NUMBER:status}
- Overwrite Existing Fields:
true
Result
Input:
message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",
Output:
ip: "1.1.1.2"
path: "/"
timestamp: "11/June/2023:03:04:55·+0100"
status: "200"
request_method: "GET"
message: "1.1.1.2 [11/June/2023:03:04:55 +0100] \"GET /\" 200",
Go Regex example
This example uses Go regular expressions (regexp) to extract matching fields from an event field. The regex101 site is useful for testing Go regular expressions.
Configuration
- Field Name:
message
- Expression Type:
goregex
- Expression:
RequestId: (?P<request_id>[-0-9a-z]+)[ ]+Duration: (?P<duration>[.0-9]+) ms
- Overwrite Existing Fields:
true
Result
Input:
message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95 Duration: 15.74 ms"
Output:
message: "REPORT RequestId: 8f507cfc-xmpl-4697-b07a-ac58fc914c95 Duration: 15.74 ms"
request_id: "8f507cfc-xmpl-4697-b07a-ac58fc914c95",
duration: "15.74"
Example Expressions
Below are some examples of how to construct regular expression with capture groups in Go and Grok. In these examples we've constructed expressions to parse CLF log messages. For more information on constructing expression view the Go regular expressions and Grok expression docs linked above.
Go regular expression to parse CLF
(?P<ip>.*?) (?P<remote_log_name>.*?) (?P<userid>.*?) \[(?P<date>.*?) (?P<timezone>.*?)\] \"(?P<request_method>.*?) (?P<path>.*?)(?P<request_version> HTTP/.*)?\" (?P<status>.*?) (?P<length>.*?) \"(?P<referrer>.*?)\" \"(?P<user_agent>.*?)\" (?P<session_id>.*?) (?P<generation_time_micro>.*?) (?P<virtual_host>.*)
Grok regular expression to parse CLF
%{IP:ip} %{DATA:remote_log_name} %{DATA:userid} \[%{HTTPDATE:date}\] \"%{WORD:request_method} %{DATA:path}\" %{NUMBER:status} %{NUMBER:length} \"%{DATA:referrer}\" \"%{DATA:user_agent}\" %{DATA:session_id} %{NUMBER:generation_time_micro} %{GREEDYDATA:virtual_host}