How to get Started With FluentBit

About

FluentBit ¹⁾ from Calyptia is a observability pipeline tool (written in C, that works on Linux and Windows).

It's the Fluentd successor with smaller memory footprint ²⁾

Steps

Parser

When you need to parse log file, you need to define their format via a Parser format.

The parser format are defined in a parser file ³⁾

Regex Parser definition

The regex parser definition is based on named regular expression group

Example: in the parser definition, the regex key has the regular expression

[PARSER]
    Name myparser
    Format regex
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$

where each regular expression group follows the same pattern:

(?<name>): define the name of the group (ie the name of the field)
[^ ]+: is a negative class (due to the ^) that will capture one or more characters until it find a space (the only character between the brackets).

Available Regex Parser definition

The FluentBit repository has already example of regexp parser files that you can use to build your own parser file.

The files are available in the conf github directory and starts with parsers

Example: Extract of the parsers.conf file For:

the Apache log

[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

the Nginx log

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

¹⁾

FluentBit

²⁾

https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit

³⁾

FluentBit file are configuration file with their own format specified here