(Stream|Pipe|Message Queue|Event Processing)
About
From an abstract point of view, a stream is a sequence of an infinite cardinality (size) delivered at unknown time intervals.
An finite sequence is called a list
Streams:
- are inputs and outputs of operations
Operations:
- functional-style operations on streams of elements on collections, such as map-reduce transformations.
Collections are primarily concerned with the efficient management of, and access to, their elements. By contrast, streams do not provide a means to directly access or manipulate their elements, and are instead concerned with declaratively describing their source and the computational operations which will be performed in aggregate on that source.
To perform a computation, stream operations are composed into a stream pipeline. A stream pipeline can be viewed as a query on the stream source.
Because processing of stream is also infinite, streams are associated to realtime processing.
All data processing algorithm cannot rely on the size to make assumption.
System that manages a stream are called messaging system.
Stream processing lets model systems that have state without ever using assignment or mutable data.
The data structures involved in streams application are then:
An application that handle this message is called a messaging system.
In a stream architecture, stream processing is using the observer operator:
- Something happened (A new element in the stream such as an Event),
- Subscribe to it (Streams)
A table is a stream of data manipulation with an infinite windows.
The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail.
Event sourcing describes a process as a sequence of event.
Articles Related
Definition
A stream is a infinite sequence of element delivered at unknown time intervals.
A stream is derivative of state over time. The product rule, (uv)' = u'v + uv', is analogous to the rule for joining streams.
Streaming concepts
- characteristics of unbounded streams,
- time,
- and state
Architecture
A messaging technology needs to have the following characteristics:
- Replayable
- Persistent
- Capable of high performance at large scale
Vision
| Real-time Mapreduce | Event-driven microservices |
|---|---|
| Storm, Spark Streaming, Flink | Kafka Stream API |
| Central cluster | Embedded library in any Java app |
| Custom packaging, deployment & monitoring | Just Kafka and your app |
| Suitable for analytics-type use cases | Makes stream processing accessible to any use case |
Event Centric
Example
Streams of data
- user activity on a website
- sensor readings from devices (IOT)
- order delivery
- …