# Shuffle

Glossary, read 10 Nov 2026

**The short answer.** A shuffle is Spark's mechanism for redistributing data so that it is grouped differently across partitions. It copies data between executors and machines, which the Spark programming guide calls a complex and costly operation. Joins, group-bys and repartitions trigger shuffles, and spark.sql.shuffle.partitions sets 200 output partitions by default.

Shuffle cost has three parts: serializing and writing map output to local disk, moving it over the network, and reading and merging it on the other side. Disk and network are why shuffle-heavy jobs slow down on small or throttled instances.

Several accelerators replace parts of this path. Gluten and Comet ship their own columnar shuffle manager, and remote shuffle services such as Celeborn move it off the executors.

## A sourced example

The Spark RDD guide uses reduceByKey as its example: all values for one key must sit on the same partition, so Spark reads from every partition and moves the matching rows together.

## Sources

- Apache Spark RDD programming guide: Shuffle operations: https://spark.apache.org/docs/latest/rdd-programming-guide.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/shuffle/
