(Stream|Pipe|Message Queue|Event Processing)

About

From an abstract point of view, a stream is a sequence of an infinite cardinality (size) delivered at unknown time intervals.

An finite sequence is called a list

Streams:

  • are inputs and outputs of operations
  • may be also buffers (samza) or not (Java)

Operations:

Collections are primarily concerned with the efficient management of, and access to, their elements. By contrast, streams do not provide a means to directly access or manipulate their elements, and are instead concerned with declaratively describing their source and the computational operations which will be performed in aggregate on that source.

To perform a computation, stream operations are composed into a stream pipeline. A stream pipeline can be viewed as a query on the stream source.

Because processing of stream is also infinite, streams are associated to realtime processing.

All data processing algorithm cannot rely on the size to make assumption.

System that manages a stream are called messaging system.

Stream processing lets model systems that have state without ever using assignment or mutable data.

The data structures involved in streams application are then:

An application that handle this message is called a messaging system.

In a stream architecture, stream processing is using the observer operator:

  • Something happened (A new element in the stream such as an Event),
  • Subscribe to it (Streams)

A table is a stream of data manipulation with an infinite windows.

The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail.

Event sourcing describes a process as a sequence of event.

Definition

A stream is an infinite sequence of element delivered at unknown time intervals.

A stream is derivative of state over time. The product rule, (uv)' = u'v + uv', is analogous to the rule for joining streams.

Streaming concepts

  • characteristics of unbounded streams,
  • time,
  • and state

Architecture

A messaging technology needs to have the following characteristics:

  • Replayable
  • Persistent
  • Capable of high performance at large scale

Vision

Real-time Mapreduce Event-driven microservices
Storm, Spark Streaming, Flink Kafka Stream API
Central cluster Embedded library in any Java app
Custom packaging, deployment & monitoring Just Kafka and your app
Suitable for analytics-type use cases Makes stream processing accessible to any use case

Event Centric

Event Centric Thinking

Example

Streams of data

  • user activity on a website
  • sensor readings from devices (IOT)
  • order delivery

Documentation / Reference

Task Runner