Data Processing - (Pipeline | Compose | Chain)

1 - About

A pipeline is a finite automata where:

A pipeline creates a composition relationship.

A pipeline is also known as:

  • Compose
  • Chain (for instance Chain of command) - Daisy Chain ;)
A dataflow (data workflow) and pipeline are generally synonym
But a pipeline follows more a compositional structure (cascade of operations) whereas a data flow shows a directed graph structure (loop may be present for instance)

3 - Type

3.1 - Imperative

The pipeline is executed step by step

3.2 - Declarative

The pipeline is executed only when the terminal operation is called.

All steps are building a composite type known as algebraic data type.

4 - Example

4.1 - Shell

In an OS Shell (Dos, bash), a serie of command connected by the pipe operators forms a pipeline. See Shell Data Processing - Pipeline

4.2 - Code

By returning the calling object from a function, you can compose (or chain) functions. See Design Pattern - (Object) Builder. When we compose (chain) an operation, the output of one operation becomes the input for the next operation, and operations are applied from left to right.

4.3 - Message / Queue

4.4 - MapReduce

5 - Library

6 - Documentation / Reference

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap