Skip to content
Spark Acceleration Compare

Glossary, read 10 Nov 2026

Spill

All termsOpen the Ledger

Spill is a symptom, not a setting. It usually points to partitions that are too large, skewed keys, or executors with too little memory per task. Fixing it by hand means changing partition counts and memory ratios.

Some vendors say their option reduces or removes spill. Those are vendor claims; read them against the Spark UI numbers from a real job.

A sourced example

The Spark configuration reference lists spark.shuffle.spill.diskWriteBufferSize, default 1024 * 1024 bytes (1 MiB), the buffer used when writing sorted records out to the on-disk spill file.

Sources

  1. Apache Spark documentation: Web UI, Stages tab, read 10 Nov 2026

Last reviewed 10 Nov 2026.