Data Management - (Transaction|Request|Commit|Redo) Log

1 - About

(Transaction|Request|commit) logs are structured log file store all changes made to the data as they occur.

They permits the implementation of :

same as Transactions - Write-Ahead Logging (Rollback journal) - WAL ?

See Design Pattern - (Command|Action|Transaction)

A transaction log is made up of entries (also records), which record all changes made to data.

This log are also known as:

The truth is the log. The database is a cache of a subset of the log.

If we keep events in a transaction log, it starts to behave like a version control system for data (versus code).

Most databases write every single insert, update and delete operation to a transaction log. This serves as a source of truth to the database and in case of errors, the database state is recovered from them.

3 - Database Usage

If have all your logs file, you can reconstruct the entire database

4 - Distributed

Application Purpose Machine Replication (High Availability)
Kafka (Event Hub) Publish-subscribe messaging rethought as a distributed commit log

5 - Management

5.1 - Transfer them

How to transfer log files into a log manager such as Kafka

  • Kafka Connect
  • Logstash
  • FileBeats from Elastic
  • Log Tailers
  • Develop an Application's logger that send straight to (No files involved at all).
  • Telegraf (for metrics oriented messages)

5.2 - Read them

6 - Implementation

7 - Documentation / Reference

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap