Glossary, read 10 Nov 2026
Vectorized execution
Spark's Parquet reader hands over columnar batches at the scan. Native engines say they keep data columnar through more of the plan, including shuffle.
Batch size is a tradeoff. Larger batches amortize overhead but raise memory use and out-of-memory risk, which the Spark documentation says to choose carefully.
A sourced example
Spark's spark.sql.inMemoryColumnarStorage.batchSize defaults to 10000 rows for cached tables, a different setting from the 4096-row Parquet reader batch, and the two do not need to match.
Sources
- Apache Spark configuration reference, read 10 Nov 2026
Last reviewed 10 Nov 2026.