Data Processing - (Pipeline | Compose | Chain)

About

A pipeline is a finite automata where:

A pipeline creates a composition relationship.

A pipeline is also known as:

  • Compose
  • Chain (for instance Chain of command) - Daisy Chain ;)

A dataflow (data workflow) and pipeline are generally synonym
But a pipeline follows more a compositional structure (cascade of operations) whereas a data flow shows a directed graph structure (loop may be present for instance)

Type

Imperative

The pipeline is executed step by step

Declarative

The pipeline is executed only when the terminal operation is called.

All steps are building a composite type known as algebraic data type.

Example

Shell

In an OS Shell (Dos, bash), a serie of command connected by the pipe operators forms a pipeline. See Shell Data Processing - Pipeline

Code

By returning the calling object from a function, you can compose (or chain) functions. See Design Pattern - (Object) Builder. When we compose (chain) an operation, the output of one operation becomes the input for the next operation, and operations are applied from left to right.

Message / Queue

MapReduce

Library

Documentation / Reference


Powered by ComboStrap