# Partition

Glossary, read 10 Nov 2026

**The short answer.** A partition is a slice of a dataset that one Spark task processes. For file sources, spark.sql.files.maxPartitionBytes packs up to 128 MB into one partition by default. For shuffles, spark.sql.shuffle.partitions sets 200 partitions by default, and AQE can coalesce them after the stage runs.

Too many small partitions waste time on scheduling and file opens. Too few large ones cause spill, skew and idle cores.

Accelerators change what a good partition size is. NVIDIA's tuning guide lists concurrent GPU tasks as a setting, and DualBird says only a few tasks run at once on its FPGA, so settings tuned for CPU executors may not carry over.

## A sourced example

The RAPIDS Accelerator tuning guide lists input partition size and shuffle partitions as settings to revisit; in AWS's g7.4xlarge test the GPU configuration set concurrentGpuTasks to 3.

## Sources

- Apache Spark configuration reference: spark.sql.files.maxPartitionBytes: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/partition/
