Skip to content
Spark Acceleration Compare

Glossary, read 10 Nov 2026

Skew

All termsOpen the Ledger

A skewed stage runs as long as its slowest task, so the cluster sits idle while one executor finishes. Raising the cluster size does not fix it.

DualBird says skew costs it less time because few tasks run at once on its hardware. That is a vendor claim. See AQE for Spark's built-in handling.

A sourced example

The two defaults that define a skewed partition are spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes = 256MB and spark.sql.adaptive.skewJoin.skewedPartitionFactor = 5.0; both conditions must hold.

Sources

  1. Apache Spark configuration reference: adaptive skew join, read 10 Nov 2026

Last reviewed 10 Nov 2026.