Data Concurrency - Producer Consumer Thread

1 - About

Producer / Consumer is concurrency model (ie two threads/process communication) where:

  • one thread called a Producer sends data
  • and the other thread called the Consumer receive data.

The data send and received is called a message.

3 - Pipeline

When the consumer becomes also a producer for another consumer, they form a chain called a pipeline.

See also Event - Provider

3.1 - Upstream faster than downstream

What solutions do we have when there is a fast producer and a slow consumer (ie if the upstream is faster, than the downstream)

  • Backpressure - the slower consumer request a given amount of data from the faster producer
  • Buffer the messages in a queue
  • Drop some message - networking hardware does that
  • Block the producer until the consumer catch up. It slows down the entire pipeline.

Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap