Data System Architecture


LogStash is:


A Logstash pipeline is composed of the following elements;

  • input (produce the data)
  • filter (optional, process the data)
  • output (write the data)

For instance:

  • it can read a log file
  • parse it into a JSON format via a grok expression (filter)
  • send them to sink (databases, index engines and so forth).


The plugins are hosted on rubygems:

The official plugins are visible on Github: logstash-plugins github

Logstash is written with Java and Ruby is supported thanks to JRuby. Plugin may be written in Ruby or Java.

Getting Started



  • Download
docker pull docker.elastic.co/logstash/logstash:7.5.1
  • Run
docker run ^
   --rm ^
   -it ^
   -v ^
   %CD%:/usr/share/logstash/pipeline/ ^
   docker.elastic.co/logstash/logstash:7.5.1 ^


  • the %CD% is the current directory and will be mounted into /usr/share/logstash/pipeline/ which is the location of the pipeline configuration file logstash.conf. If there is no configuration, the (BeatInput) is used.

the log4 configuration file is located at /usr/share/logstash/config/log4j2.properties

First pipeline

cd bin/logstash-7.5.1
# in docker
# cd /usr/share/logstash/bin/logstash
logstash -e 'input { stdin { } } output { stdout {} }'


Typing hello word at the console (stdin) will produce the below message at the console (stdout)

       "message" => "hello world",
    "@timestamp" => 2020-01-13T14:02:43.376Z,
      "@version" => "1",
          "host" => "32621775747d"

Logstash adds timestamp and IP address information to the message.

Documentation / Reference

Recommended Pages
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...

is an extension of regular expressions that supports expressions as variables (so they can be reused) In this example, we will construct an expression that matches the part of a string time expression....
Data System Architecture
Log - Collector (Aggregator)

A log collector will collect log file, optionally transform them and deliver them to one or more destination such as: a log server or metrics server Name Log to Log Log To Metrics Description...

Share this page:
Follow us:
Task Runner