Stream - (Software|Library)

Software, Library around the notion of stream

• Dos - Pipeline
• Bash - Pipeline

Distributed stream processing frameworks such as:

• Kafka (Event Hub)
• Samza - Linkedin,
• Storm - team (Yahoo!)
• Flink
• Amazon Kinesis
• Spark Streaming. See Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher)
• StreamSet Connected car demo
• Apache Spark (Micro batch)

These streams are built into the application itself: IO - Standard streams (stdin, stdout, stderr)

Node.js stream uses the pipe function to connect the source to the destination

src.pipe(dst)

<wrap box>.pipe(dst)</note> returns <wrap box>dst</note> so that you can chain together multiple <wrap box>.pipe</note> calls together:

a.pipe(b).pipe(c).pipe(d)

which is the same as:

a.pipe(b);
b.pipe(c);
c.pipe(d);

which is the same as with the shell command-line pipe:

a | b | c | d

Java Stream (Map reduce implementation, only one source then)

int sum = widgets.stream()
                      .filter(b -> b.getColor() == RED)
                      .mapToInt(b -> b.getWeight())
                      .sum();

Stream - Interface

• pipes and filters
• stream-handbook
• https://www.mapr.com/ebooks/streaming-architecture/