Skip to content
Spark Acceleration Compare

Glossary, read 10 Nov 2026

Partition

All termsOpen the Ledger

Too many small partitions waste time on scheduling and file opens. Too few large ones cause spill, skew and idle cores.

Accelerators change what a good partition size is. NVIDIA's tuning guide lists concurrent GPU tasks as a setting, and DualBird says only a few tasks run at once on its FPGA, so settings tuned for CPU executors may not carry over.

A sourced example

The RAPIDS Accelerator tuning guide lists input partition size and shuffle partitions as settings to revisit; in AWS's g7.4xlarge test the GPU configuration set concurrentGpuTasks to 3.

Sources

  1. Apache Spark configuration reference: spark.sql.files.maxPartitionBytes, read 10 Nov 2026

Last reviewed 10 Nov 2026.