What are Log Collectors (Aggregators)?

Data System Architecture

About

A log collector will collect log file, optionally transform them and deliver them to one or more destination such as:

List

Name Type Log to Log Log To Metrics Description
Vector (Rust, DataDog) observability pipeline tool Yes Yes
Telegraph (Go, InfluxDb)
Promtail (Graphana) Log Collector Yes Yes 1) Graphana tool to insert log Loki
Logstash (Java, Elastic) Log Collector Yes Yes 2) Elastic log collector
FluentBit Log Collector ? ? Successor to FluentD
Fluentd Log Collector ? ? Example: BigQuery
Flume Log Collector ? ?

Log to Metrics means extracting metrics from application logs to be exported into a metrics server





Discover More
Card Puncher Data Processing
Computer Monitoring / Operational Intelligence / Real Time Monitoring

Computer Monitoring / Operational Intelligence / Real Time Monitoring Monitoring is the process of defining metrics and alerts in order to respond to a performance degradation where the acceptable level...
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...
Data System Architecture
Flume - Log (Aggregator|Collector)

Apache Flume is a log collector to collect, aggregate, and move log data from many different sources to a centralized data store. aggregation = sink Apache Flume example with HTTP channel to...
Scale Counter Graph
How to get Started With FluentBit

FluentBit from Calyptia is a log collector (ie observability pipeline tool) (written in C, that works on Linux and Windows). It's the Fluentd successor with smaller memory footprint When you need...
Data System Architecture
Log - Logging

A log is a text file with: a timed list of message (activities) that an application has performed execution/request log: web server: web log ( - that stores the Http request error cron...
Data System Architecture
Log - Server

A log server is an application that is aimed to: receive log via a collector analyse them report on them and send alert if needed They are mostly search engine where: the words are stored...
Data System Architecture
LogStash

LogStash is: * a metrics collector * a log collector. * with pipeline ability A Logstash pipeline is composed of the following elements; * input (produce the data) * filter (optional, process...
Scale Counter Graph
Metrics Server

This page is software that can store and visualize counter (metrics. The collectors that sends data to a metrics server are: Metrics Collector Log Collector Instrumentation Reporter The...
Card Puncher Data Processing
Monitoring Platform

Monitoring platform provides one or more monitoring services such as: Metrics management Log management Trace Management and other monitoring service such as visualization and ticketing. They...
Scale Counter Graph
What are Application Metrics? ie Perfcounter, Performance Metrics, Operational data, Monitoring, telemetry

This section is about the collection and calculation of metrics in a monitoring context known as observability.



Share this page:
Follow us:
Task Runner