Data Processing - Lambda Architecture (batch and stream processing)

Card Puncher Data Processing


Nathan Marz wrote a blog post describing the Lambda Architecture: How to beat the CAP theorem 1)

Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both:


This approach to architecture attempts to balance:

  • latency,
  • throughput,
  • and fault-tolerance

by using:

  • batch processing to provide comprehensive and accurate views of batch data,
  • real-time stream processing to simultaneously provide views of online data.

Diagram Of Lambda Architecture Generic

Documentation / Reference

Discover More
Card Puncher Data Processing
Data Integration - Methods / Design Pattern

With multiple applications in your IT infrastructure reading and writing to and from different data stores in varying formats, it is imperative to implement a process that will let you integrate the data...
Martin Kleppmann Data Hierarchy Of Needs
Data Processing - Architecture

Fault tolerance Parallelism High Latency Delivery semantics Operations and monitoring Schema management forward-compatible data architecture: the ability to add more applications that need...
Stream Vs Batch
Stream vs Batch

This article talks Stream Processing vs Batch Processing. The most important difference is that: in batch processing the size (cardinality) of the data to process is known whereas in a stream processing,...

Share this page:
Follow us:
Task Runner