How to get Started With FluentBit

Scale Counter Graph

About

FluentBit 1) from Calyptia is a log collector (ie observability pipeline tool) (written in C, that works on Linux and Windows).

It's the Fluentd successor with smaller memory footprint 2)

Steps

Parser

When you need to parse log file, you need to define their format via a Parser format.

The parser format are defined in a parser file 3)

Regex Parser definition

The regex parser definition is based on named regular expression group

Example: in the parser definition, the regex key has the regular expression

[PARSER]
    Name myparser
    Format regex
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$

where each regular expression group follows the same pattern:

  • (?<name>): define the name of the group (ie the name of the field)
  • [^ ]+: is a negative class (due to the ^) that will capture one or more characters until it find a space (the only character between the brackets).

Available Regex Parser definition

The FluentBit repository has already example of regexp parser files that you can use to build your own parser file.

The files are available in the conf github directory and starts with parsers

Example: Extract of the parsers.conf file For:

[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z





Discover More
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...
Data System Architecture
What are Log Collectors (Aggregators)?

A log collector will collect log file, optionally transform them and deliver them to one or more destination such as: a log server or metrics server Name Type Log to Log Log To Metrics Description...
Scale Counter Graph
What are observability data pipeline tools? known as Telemetry agent

An observability data pipeline is a pipeline tool dedicated to observability data (logs, metrics, and traces). It collects them (log collector, metrics collector, ...) from multiple sources transform,...



Share this page:
Follow us:
Task Runner