Glossary, read 10 Nov 2026
Skew
A skewed stage runs as long as its slowest task, so the cluster sits idle while one executor finishes. Raising the cluster size does not fix it.
DualBird says skew costs it less time because few tasks run at once on its hardware. That is a vendor claim. See AQE for Spark's built-in handling.
A sourced example
The two defaults that define a skewed partition are spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes = 256MB and spark.sql.adaptive.skewJoin.skewedPartitionFactor = 5.0; both conditions must hold.
Sources
- Apache Spark configuration reference: adaptive skew join, read 10 Nov 2026
Last reviewed 10 Nov 2026.