Glossary, read 10 Nov 2026
Spill
Spill is a symptom, not a setting. It usually points to partitions that are too large, skewed keys, or executors with too little memory per task. Fixing it by hand means changing partition counts and memory ratios.
Some vendors say their option reduces or removes spill. Those are vendor claims; read them against the Spark UI numbers from a real job.
A sourced example
The Spark configuration reference lists spark.shuffle.spill.diskWriteBufferSize, default 1024 * 1024 bytes (1 MiB), the buffer used when writing sorted records out to the on-disk spill file.
Sources
- Apache Spark documentation: Web UI, Stages tab, read 10 Nov 2026
Last reviewed 10 Nov 2026.