#catalyst

Spark DataFrames and Spark SQL: Schema, DDL, and the Catalyst Optimizer

Catalyst turns your DataFrame ops into an optimized physical plan — understanding it explains why column pruning and predicate pushdown are free