Data Management - (Transaction|Request|Commit|Redo) Log

Data System Architecture


(Transaction|Request|commit) logs are structured log file store all changes made to the data as they occur.

They permits the implementation of :

same as Transactions - Write-Ahead Logging (Rollback journal) - WAL ?

See Design Pattern - (Command|Action|Transaction)

A transaction log is made up of entries (also records), which record all changes made to data.

This log are also known as:

The truth is the log. The database is a cache of a subset of the log.

If we keep events in a transaction log, it starts to behave like a version control system for data (versus code).

Most databases write every single insert, update and delete operation to a transaction log. This serves as a source of truth to the database and in case of errors, the database state is recovered from them.

Database Usage

If have all your logs file, you can reconstruct the entire database


Application Purpose Machine Replication (High Availability)
Kafka (Event Hub) Publish-subscribe messaging rethought as a distributed commit log


Transfer them

How to transfer log files into a log manager such as Kafka

  • Kafka Connect
  • Logstash
  • FileBeats from Elastic
  • Log Tailers
  • Develop an Application's logger that send straight to (No files involved at all).
  • Telegraf (for metrics oriented messages)

Read them


Documentation / Reference

Discover More
Data System Architecture
(Data Type | Data Structure)

A type is the classification of value according to: how they are represented (the structure) the properties that they have (the operations) A data structure is a complex data type that can not be...
Card Puncher Data Processing
Ad - Log (Web Log)

An ad server produces two types of log files: impression logs (Every time the server displays an advertisement to a customer, it adds an entry to the impression log) and click logs (Every time a customer...
Scale Counter Graph
Counter - Collector

Metrics collector query and collects metrics in order to be able to send them to a metrics server Log Collector In a instrumented application, reporter are a client piece of code which: process...
Data System Architecture
Data - History (Versioning) - Historical Data

Data history (or versioning) is just the derivation of data state. It can be found in the following data structure: where a new version of data is created by committing changes. a serie of...
Card Puncher Data Processing
Data Processing - Replication

Replication: Having a copy of the same data on multiple machines (nodes) in order to increase : Feature Example Performance serve reads in parallel, distributing application workloads across multiple...
Data System Architecture
Database - Backup

Example: Full backups every night Transaction logs every 15 minutes (going back 4 days at all times). Restore the database in the stagging system ?
Data System Architecture
Database - Migration (Deployment|Versioning|Change Management)

with SQL database. Database migration is the process of changing the structure of the database in an other with mostly DDL statement. The flow sometimes will block any change that risks data-loss. See...
Card Puncher Data Processing
Design Pattern - (Command|Action|Transaction)

Encapsulate a request as an object, letting parametrize clients with different requests, queue or log request and support undoable operations. Netflix/Hystrix/wikiNetflix Hystrix: Wrapping all...
Yarn Hortonworks
HDFS - EditLog (transaction log)

The NameNode uses a transaction log called the EditLog to persistently record every change that occurs to file system metadata. The NameNode to insert a record into the EditLog when a new file...
Card Puncher Data Processing
Information system design - Source of truth (SOT)

Source of truth. See also: Single Source of thruth In service-oriented systems there is no single source of truth, there are many. A is the authoritative data source for a given piece of information....

Share this page:
Follow us:
Task Runner