Spark - Engine


In Spark, the Spark engine is a SQL’s optimized execution engine and understand as input:

and is therefore sometimes known as the SQL Engine.

In Spark, functions are pipelined around their data structure.

A function will take a Spark data structure (data type) as input and output the same data structure (data type).

It results of a builder pattern where the last method performs an action whereas the previous one set parameters.

Spark Query Plan Generation


You are not manipulating the engine directly but via interfaces:

Documentation / Reference

Task Runner