Glossary, read 10 Nov 2026
Adaptive Query Execution
AQE waits until a shuffle stage finishes, reads the real partition sizes, and then rewrites the rest of the plan. That is why a plan shown by EXPLAIN before the run can differ from the one the Spark UI shows afterward.
Native engines that plug into Spark inherit this behavior only for the parts of the plan they leave to Spark. When you compare options, check whether the benchmark baseline ran with AQE on. See also skew and broadcast hash join.
A sourced example
The Spark configuration page lists spark.sql.adaptive.localShuffleReader.enabled (default true): after AQE turns a sort-merge join into a broadcast-hash join, Spark reads the shuffle files locally instead of over the network.
Sources
Last reviewed 10 Nov 2026.