Grok

A time variable expression

In this example, we will construct an expression that matches the part of a string time expression.

The below statements assign in grok:

• the MONTHDAY variable
• to the expression (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

With the same syntax, we can define the month number and the year

MONTHNUM (?:0?[1-9]|1[0-2])
YEAR (?>\d\d){1,2}

Now that we have defined the part of our time string, we can reuse the previous variable to create a compound expression

DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}

Web Log

In this example, we show an expression used to parse a web log (ie web server request log).

Example: The below line is an example

55.3.244.1 GET /index.html 15824 0.043

where there is:

• the request ip address
• the method
• the URL path
• the size in bytes of the response
• the duration

To parse this line, the below grok expression can be used.

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

where:

• the ip is matched with the IP base pattern and will get the variable name client
• the method is matched with the WORD pattern (ie word) and will get the variable method
• and so on

The output will be:

• client: 55.3.244.1
• method: GET
• request: /index.html
• bytes: 15824
• duration: 0.043

Standard Pattern

%{patternName:variableName[:type]}

where:

• patternName is the build-in pattern name of the regular expression pattern (ie logstash-patterns-core)
• variableName is the variable name.
• type is an optional data type conversion (by default, the type is text but logstash supports also:
  • int for integer number
  • and float for float point number

Custom Pattern

When the pattern does not exist in the standard_pattern, you can create your own expression ²⁾ by creating a named-capturing group expression.

Example:

 (?<variableName>the pattern here)

where variableName is the variable name

Custom Pattern File

If you want to be able to reuse your expression, you can create a custom pattern file in the form

patternName regularExpression

Example:

MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

You can see the built-in pattern file at logstash-patterns-core repository.