The optimizer is a program that takes a relational expression (query plan) and rewrites it with optimization rules. The output is still a relational expression and is generally called the physical plan.
The optimizer uses for that:
They are all defined in the environment object RelOptCluster.
The Planner rules transform expression trees using mathematical identities that preserve semantics.
Calcite optimizes queries by repeatedly applying planner rules to a relational expression. For example, filter early.
A cost model guides the process, and the planner engine generates an alternative expression that has the same semantics as the original but a lower cost.
Calcite optimize the query plan (ie rewrite it) in the Prepare statement class. See optimize function (L133)