About
Grok is an extension of regular expressions that supports expressions as variables (so they can be reused)
Example
A time variable expression
In this example, we will construct an expression that matches the part of a string time expression.
The below statements assign in grok:
- the MONTHDAY variable
- to the expression (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
Every (?:___) means that this is not a capturing group
With the same syntax, we can define the month number and the year
MONTHNUM (?:0?[1-9]|1[0-2])
YEAR (?>\d\d){1,2}
Now that we have defined the part of our time string, we can reuse the previous variable to create a compound expression
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
Web Log
In this example, we show an expression used to parse a web log (ie web server request log).
Example: The below line is an example
55.3.244.1 GET /index.html 15824 0.043
where there is:
To parse this line, the below grok expression can be used.
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
where:
- the ip is matched with the IP base pattern and will get the variable name client
- the method is matched with the WORD pattern (ie word) and will get the variable method
- and so on
The output will be:
- client: 55.3.244.1
- method: GET
- request: /index.html
- bytes: 15824
- duration: 0.043
Usage
Grok patterns are used to extract information from log files.
The data is then used to create telemetry metrics such as in prometheus exporter
Syntax
This section is about the syntax of a grok expression. Grok was first introduced by logstash. 1).
In a grok expression, you may use:
- a standard_pattern (built-in pattern)
- a custom_pattern (your own)
Standard Pattern
%{patternName:variableName[:type]}
where:
- patternName is the build-in pattern name of the regular expression pattern (ie logstash-patterns-core)
- variableName is the variable name.
- type is an optional data type conversion (by default, the type is text but logstash supports also:
- int for integer number
- and float for float point number
Custom Pattern
When the pattern does not exist in the standard_pattern, you can create your own expression 2) by creating a named-capturing group expression.
Example:
(?<variableName>the pattern here)
where variableName is the variable name
Custom Pattern File
If you want to be able to reuse your expression, you can create a custom pattern file in the form
patternName regularExpression
Example:
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
You can see the built-in pattern file at logstash-patterns-core repository.
Grok Debug / Editor App
If you want to test your grok expression, see:
- the grokdebug app.
Usage in Software
Filter
This section lists the known filter operations that implements Grok.