Log - Collector (Aggregator)

Data System Architecture


A log collector will collect log file, optionally transform them and deliver them to one or more destination such as:


Name Log to Log Log To Metrics Description
Vector (Rust) Yes Yes observability pipelines
Promtail Yes Yes 1) Graphana tool to insert log Loki
Logstash (Java) Yes Yes 2) Elastic log collector
Fluentd ? ? Example: BigQuery
Flume ? ?

Log to Metrics means extracting metrics from application logs to be exported into a metrics server

Recommended Pages
Scale Counter Graph
Application Metrics (Perfcounter | Performance Metrics | Operational data | Monitoring )

Application Metrics (Perfcounter | Performance Metrics | Operational data | Monitoring ) This section is about the collection and calculation of metrics in the monitoring context. These (counter|numbers)...
Card Puncher Data Processing
Computer Monitoring / Operational Intelligence / Real Time Monitoring

Computer Monitoring / Operational Intelligence / Real Time Monitoring Monitoring is the process of defining metrics and alerts in order to respond to a performance degradation where the acceptable level...
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...
Data System Architecture
Flume - Log (Aggregator|Collector)

Apache Flume is a log collector to collect, aggregate and move log data from many different sources to a centralized data store. aggregation = sink Apache Flume example with HTTP channel to...
Data System Architecture
Log - Logging

Log - Logging A log is a text file with: a timed list of message (activities) that an application has performed execution/request log: web server: web log ( - that stores the Http request...
Data System Architecture
Log - Server

A log server is an application that is aimed to: receive log via a collector analyse them report on them and send alert if needed They are mostly search engine where: the words are stored...
Data System Architecture

LogStash LogStash is: * a metrics collector * a log collector. * with pipeline ability A Logstash pipeline is composed of the following elements; * input (produce the data) * filter...
Scale Counter Graph
Metrics Server

Metrics Server This page is software that can store and visualize counter (metrics. The collectors that sends data to a metrics server are: Metrics Collector Log Collector Instrumentation...
Card Puncher Data Processing
Monitoring Platform

Monitoring platform provides one or more monitoring services such as: Metrics management Log management Trace Management and other monitoring service such as visualization and ticketing. They...

Share this page:
Follow us:
Task Runner