Glossary, read 10 Nov 2026
Whole-stage code generation
Before it, Spark 1.x used the Volcano iterator model, where each operator returned one row per call. Fusing the operators removed most of that overhead for filters, aggregates and hash joins.
Native engines such as Velox take the idea further by processing batches of columns with vectorized code. See Tungsten and vectorized execution.
A sourced example
The 2016 post measured single-thread cost per row in Spark 1.6 against 2.0: filter 15 ns to 1.1 ns, sum without group 14 ns to 0.9 ns, Parquet decoding of one int column 120 ns to 13 ns.
Sources
- Databricks blog: Apache Spark as a Compiler (May 23, 2016), read 10 Nov 2026
Last reviewed 10 Nov 2026.