Glossary, read 10 Nov 2026
Partition
Too many small partitions waste time on scheduling and file opens. Too few large ones cause spill, skew and idle cores.
Accelerators change what a good partition size is. NVIDIA's tuning guide lists concurrent GPU tasks as a setting, and DualBird says only a few tasks run at once on its FPGA, so settings tuned for CPU executors may not carry over.
A sourced example
The RAPIDS Accelerator tuning guide lists input partition size and shuffle partitions as settings to revisit; in AWS's g7.4xlarge test the GPU configuration set concurrentGpuTasks to 3.
Sources
- Apache Spark configuration reference: spark.sql.files.maxPartitionBytes, read 10 Nov 2026
Last reviewed 10 Nov 2026.