# What to do when Spark tuning is not enough

Guide, read 10 Nov 2026

**The short answer.** Spark performance tuning costs engineer time, not new hardware: adaptive query execution (on by default since Spark 3.2.0), partition sizing, broadcast joins, file sizes and skew settings. This guide lists each documented default and the signs in the Spark UI that settings alone no longer move the bill.

## What tuning changes, and what it needs

Tuning changes configuration and file layout. It needs no new plugin, no new instance type and no vendor contract, and each change can be reverted. A tuned job is the baseline a trial compares against. It also changes the answer to the question every accelerator vendor gets: faster than what? A baseline that still runs 200 untuned shuffle partitions over tiny files makes any engine look fast.

This guide lists the settings the Apache Spark documentation names, with the defaults it publishes, then says how to tell when you have run out of room. It is not a promise that any setting helps your jobs.

## The checklist, with documented defaults

| Area | Setting (default per Spark docs) | What to check |
| --- | --- | --- |
| Adaptive query execution | spark.sql.adaptive.enabled (true since Spark 3.2.0) | Confirm the setting is on in the cluster configuration; Spark 3.2.0 made it the default |
| Shuffle partitions | spark.sql.shuffle.partitions (200); AQE advisory size 64 MB | With AQE on, Spark coalesces small partitions toward the advisory size; set the initial number high enough |
| Broadcast joins | spark.sql.autoBroadcastJoinThreshold (10 MB) | Dimension tables a little over the threshold run as sort-merge joins; check the SQL tab for the join Spark chose |
| File sizes on read | spark.sql.files.maxPartitionBytes (128 MB); openCostInBytes (4 MB) | Thousands of files under a few MB each make scan time a listing and open problem |
| Skew | spark.sql.adaptive.skewJoin.skewedPartitionFactor (5.0); threshold 256 MB | A partition is skewed only if it is above both the factor times the median and the byte threshold |
| Serialization | spark.serializer (Java); Kryo recommended by the guide for network-heavy jobs | Kryo needs class registration; the tuning guide says often as much as 10x faster |
| Parallelism | Tuning guide: 2 to 3 tasks per CPU core | Tasks as short as 200 ms are efficient per the guide |

Defaults read 2026-11-10 from the Apache Spark SQL performance tuning page and the Tuning Spark page.

## Adaptive query execution, partition sizing and broadcast joins

AQE re-plans the query at runtime from statistics gathered at each shuffle. It does three things the Spark documentation describes: it coalesces contiguous shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB), it switches a sort-merge join to a broadcast join when a side turns out small, and it splits skewed partitions in a sort-merge join. Spark 3.2.0 turned it on by default, so on a current runtime the work is checking that nobody turned it off and that spark.sql.shuffle.partitions is not set so low that the coalescing has nothing to start from. Amazon EMR documents AQE, dynamic partition pruning, Bloom filter join and optimized join reorder in its performance-optimized runtime; the AQE-related features are on by default.

For broadcast joins, the default of 10 MB is conservative for modern executors. Raising it makes more joins skip the shuffle, but the broadcast table must fit in driver and executor memory, and the timeout is 300 seconds by default (spark.sql.broadcastTimeout). A join hint (BROADCAST, MERGE, SHUFFLE_HASH) overrides the choice for one query, with BROADCAST taking priority over the others per the documentation.

## File sizes and skew

Scan cost depends on layout. Spark packs files into read partitions up to 128 MB, charging 4 MB of estimated cost per file opened. A table of many 1 MB files costs more to read than a table of the same bytes in 128 MB files. Compaction jobs fix that, and the Iceberg compaction bill is a separate cost for teams that run it daily. The Spark tuning guide also says to raise parallelism when reduce tasks run out of memory because each handles too much, and notes that listing a very large number of input directories against an object store such as S3 can take a long time.

Skew shows up as one task far longer than its siblings. AQE's skew handling only splits partitions above 256 MB that are also more than 5.0 times the median partition size. A skewed key below those thresholds, or skew in a group-by instead of a join, stays. The fixes then are salting a key or filtering the hot value out and handling it separately, which are code changes.

## How to tell tuning is not enough

Look for these signs in the Spark UI after the settings above are applied. The stage with the most task time is dominated by CPU, not by waiting on a shuffle fetch or a single straggler. Spill is small, and GC time is a small share of task time. Input size and file counts are reasonable. If those hold and the job is still slow or the bill still rising, the remaining cost is the engine's own execution speed, which is what the options on this site change.

The [two-week trial](/guides/trialing-an-accelerator/) guide uses the tuned job as its baseline. The share of [SQL and DataFrame work](/guides/what-no-code-change-means/) against UDF and RDD code matters, because those stages stay on the CPU in every plugin documented here.

## What DualBird says about tuning

DualBird claims that tens of Spark settings (executor sizing, CPU-to-memory ratios, parallelism) become irrelevant on its path, that disk spills effectively disappear, and that cluster sizing reduces to the number of F2 nodes. A later post in its series says skew still costs time, but less, and needs no repartitioning or salting. Those are DualBird's claims and it publishes no independent test of them.

Those claims hold only on AWS F2 instances, which leaves out Azure, Google Cloud and Databricks-managed clusters, and every DualBird benchmark so far is its own.

## Sources

- Apache Spark documentation: SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026)
- Apache Spark documentation: Tuning Spark: https://spark.apache.org/docs/latest/tuning.html (read 10 Nov 2026)
- Apache Spark documentation: Configuration: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026)
- Amazon EMR documentation: Spark performance features: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026)
- DualBird product page: https://dualbird.io/product (read 10 Nov 2026)
- DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/when-tuning-is-cheaper/
