About
(Transaction|Request|commit) logs are structured log file store all changes made to the data as they occur.
They permits the implementation of :
- fault tolerance (Undo/Redo)
same as Transactions - Write-Ahead Logging (Rollback journal) - WAL ?
See Design Pattern - (Command|Action|Transaction)
A transaction log is made up of entries (also records), which record all changes made to data.
This log are also known as:
- Transaction log
- Request log
- commit log
- redo log
The truth is the log. The database is a cache of a subset of the log.
If we keep events in a transaction log, it starts to behave like a version control system for data (versus code).
Most databases write every single insert, update and delete operation to a transaction log. This serves as a source of truth to the database and in case of errors, the database state is recovered from them.
Articles Related
Database Usage
- Recover consistent state of a database
- Replicate the database (Dataguard, Streams, GoldenGate…)
- Update materialized view
If have all your logs file, you can reconstruct the entire database
Distributed
| Application | Purpose |
|---|---|
| http://bookkeeper.apache.org/ | Machine Replication (High Availability) |
| Kafka (Event Hub) | Publish-subscribe messaging rethought as a distributed commit log |
Management
Transfer them
How to transfer log files into a log manager such as Kafka
- Kafka Connect
- Logstash
- FileBeats from Elastic
- Log Tailers
- Develop an Application's logger that send straight to (No files involved at all).
- Telegraf (for metrics oriented messages)
Read them
Implementation
Documentation / Reference
- The Log: What every software engineer should know about real-time data's unifying abstraction (What is a log, Log in Database, Log in Distributed database)