Spark Engine - Optimizer (Catalyst)

Spark Query Plan Generation

About

SQL Engine - Query Optimizer (Query Optimization)

Example

A map and a filter can be performed in one operation.

Documentation / Reference




Discover More
Card Puncher Data Processing
Calcite - Planner (RelOptPlanner)

org/apache/calcite/plan/package-infoPlan provides an optimizer interface (ie Defines interfaces for constructing rule-based optimizers of relational expressions) Frameworks...
Spark Query Plan Generation
Spark Engine - (Operations | Functions )

Operations are divided into transformations and actions. Transformations are pipelined function (producing the same input type), and actions trigger computation and return results. Transformation functions...
Spark Query Plan Generation
Spark Engine - Logical Plan

Logical Plan in Spark. Each data structure represents a logical plan that describes the computation required to produce the data. When an action is invoked, Spark's query optimizer optimizes the...
Spark Query Plan Generation
Spark Engine - Physical Plan

in Spark When an action is invoked, Spark's query optimizer optimizes the logical plan and generates a physical plan for efficient execution. To explore the physical plan, use the explain...



Share this page:
Follow us:
Task Runner