Data Concurrency - Producer Consumer Thread

Data System Architecture

About

Producer / Consumer is concurrency model (ie two threads/process communication) where:

  • one thread called a Producer sends data
  • and the other thread called the Consumer receive data.

The data send and received is called a message.

Pipeline

When the consumer becomes also a producer for another consumer, they form a chain called a pipeline.

See also Event - Provider

Upstream faster than downstream

What solutions do we have when there is a fast producer and a slow consumer (ie if the upstream is faster, than the downstream)

  • Backpressure - the slower consumer request a given amount of data from the faster producer
  • Buffer the messages in a queue
  • Drop some message - networking hardware does that
  • Block the producer until the consumer catch up. It slows down the entire pipeline.





Discover More
Data System Architecture
Concurrency - Model (Design)

list of asynchronous (concurrent/parallel) design producer-consumer, messaging, promise / future
Event Centric Thinking
Data Flow - Backpressure

When the dataflow runs through asynchronous steps, each step may perform different things with different speed. In this setting, Back-pressure means a fast producer and slow consumer. The issue of...
Card Puncher Data Processing
Data Processing - Reactive Stream Processing

Reactive Streams is the standard/specification for reactive data processing (ie observer, asynchronous processing) The characteristics are: functional programming fashion non-blocking backpressure...
Java Conceptuel Diagram
Java Concurrency - (Queue|Stack)

and in Java concurrency context Interface and Implementation The java/util/concurrent/BlockingQueueBlockingQueue interface defines a first-in-first-out data structure that blocks or times out...
Java Conceptuel Diagram
Java Concurrency - Executor (Thread manager)

An executor is a thread management module (thread creation and management). java.util.concurrent provides two executor interfaces: java/util/concurrent/ExecutorExecutor is a simple standardized...
Java Conceptuel Diagram
Java Concurrency - Model

Which concurrency model should you use in @Java? Threads, Actors, Executors, Fork Join, Fibers?
Oracle Database Consumer Producer
Oracle Database - Producer / Consumer Model

The producer consumer model is used in order to execute a statement in parallel. Thesets of PX servers work in pairs: one set is producing rows (producer) and one set is consuming the rows (consumer)....
Data System Architecture
Parallel Programming - Producer Consumer problem (Bounded-buffer problem)

The producer–consumer problem (also known as the bounded-buffer problem) is a classic example of a multi-process synchronization problem in a producer consumer mode. The problem describes two processes...
Event Centric Thinking
Stream - Interface

A stream interface will expose two functions: A src function that produces the objects (file, ...) A dest function that consumes the objects (file, ...) See also:



Share this page:
Follow us:
Task Runner