Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

What to do when Spark tuning is not enough

All guidesOpen the Ledger

What tuning changes, and what it needs

Tuning changes configuration and file layout. It needs no new plugin, no new instance type and no vendor contract, and each change can be reverted. A tuned job is the baseline a trial compares against. It also changes the answer to the question every accelerator vendor gets: faster than what? A baseline that still runs 200 untuned shuffle partitions over tiny files makes any engine look fast.

This guide lists the settings the Apache Spark documentation names, with the defaults it publishes, then says how to tell when you have run out of room. It is not a promise that any setting helps your jobs.

The checklist, with documented defaults

Defaults read 2026-11-10 from the Apache Spark SQL performance tuning page and the Tuning Spark page.
AreaSetting (default per Spark docs)What to check
Adaptive query executionspark.sql.adaptive.enabled (true since Spark 3.2.0)Confirm the setting is on in the cluster configuration; Spark 3.2.0 made it the default
Shuffle partitionsspark.sql.shuffle.partitions (200); AQE advisory size 64 MBWith AQE on, Spark coalesces small partitions toward the advisory size; set the initial number high enough
Broadcast joinsspark.sql.autoBroadcastJoinThreshold (10 MB)Dimension tables a little over the threshold run as sort-merge joins; check the SQL tab for the join Spark chose
File sizes on readspark.sql.files.maxPartitionBytes (128 MB); openCostInBytes (4 MB)Thousands of files under a few MB each make scan time a listing and open problem
Skewspark.sql.adaptive.skewJoin.skewedPartitionFactor (5.0); threshold 256 MBA partition is skewed only if it is above both the factor times the median and the byte threshold
Serializationspark.serializer (Java); Kryo recommended by the guide for network-heavy jobsKryo needs class registration; the tuning guide says often as much as 10x faster
ParallelismTuning guide: 2 to 3 tasks per CPU coreTasks as short as 200 ms are efficient per the guide

Adaptive query execution, partition sizing and broadcast joins

AQE re-plans the query at runtime from statistics gathered at each shuffle. It does three things the Spark documentation describes: it coalesces contiguous shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB), it switches a sort-merge join to a broadcast join when a side turns out small, and it splits skewed partitions in a sort-merge join. Spark 3.2.0 turned it on by default, so on a current runtime the work is checking that nobody turned it off and that spark.sql.shuffle.partitions is not set so low that the coalescing has nothing to start from. Amazon EMR documents AQE, dynamic partition pruning, Bloom filter join and optimized join reorder in its performance-optimized runtime; the AQE-related features are on by default.

For broadcast joins, the default of 10 MB is conservative for modern executors. Raising it makes more joins skip the shuffle, but the broadcast table must fit in driver and executor memory, and the timeout is 300 seconds by default (spark.sql.broadcastTimeout). A join hint (BROADCAST, MERGE, SHUFFLE_HASH) overrides the choice for one query, with BROADCAST taking priority over the others per the documentation.

File sizes and skew

Scan cost depends on layout. Spark packs files into read partitions up to 128 MB, charging 4 MB of estimated cost per file opened. A table of many 1 MB files costs more to read than a table of the same bytes in 128 MB files. Compaction jobs fix that, and the Iceberg compaction bill is a separate cost for teams that run it daily. The Spark tuning guide also says to raise parallelism when reduce tasks run out of memory because each handles too much, and notes that listing a very large number of input directories against an object store such as S3 can take a long time.

Skew shows up as one task far longer than its siblings. AQE's skew handling only splits partitions above 256 MB that are also more than 5.0 times the median partition size. A skewed key below those thresholds, or skew in a group-by instead of a join, stays. The fixes then are salting a key or filtering the hot value out and handling it separately, which are code changes.

How to tell tuning is not enough

Look for these signs in the Spark UI after the settings above are applied. The stage with the most task time is dominated by CPU, not by waiting on a shuffle fetch or a single straggler. Spill is small, and GC time is a small share of task time. Input size and file counts are reasonable. If those hold and the job is still slow or the bill still rising, the remaining cost is the engine's own execution speed, which is what the options on this site change.

The two-week trial guide uses the tuned job as its baseline. The share of SQL and DataFrame work against UDF and RDD code matters, because those stages stay on the CPU in every plugin documented here.

What DualBird says about tuning

DualBird claims that tens of Spark settings (executor sizing, CPU-to-memory ratios, parallelism) become irrelevant on its path, that disk spills effectively disappear, and that cluster sizing reduces to the number of F2 nodes. A later post in its series says skew still costs time, but less, and needs no repartitioning or salting. Those are DualBird's claims and it publishes no independent test of them.

Those claims hold only on AWS F2 instances, which leaves out Azure, Google Cloud and Databricks-managed clusters, and every DualBird benchmark so far is its own.

Sources

  1. Apache Spark documentation: SQL performance tuning, read 10 Nov 2026
  2. Apache Spark documentation: Tuning Spark, read 10 Nov 2026
  3. Apache Spark documentation: Configuration, read 10 Nov 2026
  4. Amazon EMR documentation: Spark performance features, read 10 Nov 2026
  5. DualBird product page, read 10 Nov 2026
  6. DualBird: A simpler Spark, read 10 Nov 2026

Last reviewed 10 Nov 2026.