Glossary, read 10 Nov 2026
Shuffle
Shuffle cost has three parts: serializing and writing map output to local disk, moving it over the network, and reading and merging it on the other side. Disk and network are why shuffle-heavy jobs slow down on small or throttled instances.
Several accelerators replace parts of this path. Gluten and Comet ship their own columnar shuffle manager, and remote shuffle services such as Celeborn move it off the executors.
A sourced example
The Spark RDD guide uses reduceByKey as its example: all values for one key must sit on the same partition, so Spark reads from every partition and moves the matching rows together.
Sources
- Apache Spark RDD programming guide: Shuffle operations, read 10 Nov 2026
Last reviewed 10 Nov 2026.