Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

Where a Spark job spends its time

All guidesOpen the Ledger

Why the split matters before you buy anything

Every acceleration option on this site attacks one or more of the same stages. A native engine replaces the JVM operators. A GPU or FPGA takes over sort, join and aggregation. A tuning change shrinks shuffle or skips files. If you do not know where your own jobs spend time, you cannot tell which published speedup applies to you.

This guide lists the stages, what the Apache Spark documentation says about each, and the few published measurements of where time goes. It gives no pick and no estimate for your cluster. The last section says what to read in the Spark UI to get your own split.

The stages of a typical SQL job

A Spark SQL query is planned by Catalyst, cut into stages at each shuffle, and run as tasks on executor JVMs. Time in a stage divides roughly into the parts below. Each part has a metric in the Spark UI.

  • Scan and decode. Reading Parquet or ORC from S3 or HDFS, decompressing it and turning columns into rows. Spark packs files into read partitions of up to 128 MB (spark.sql.files.maxPartitionBytes), and charges 4 MB of estimated cost to open each file (spark.sql.files.openCostInBytes), which is why many small files are slow.
  • Shuffle write and read. Data is partitioned, written to local disk by map tasks and fetched over the network by reduce tasks. The default is 200 shuffle partitions (spark.sql.shuffle.partitions). The UI shows shuffle read and write size and the time spent writing shuffle data.
  • Sort and join. Sort-merge joins sort both sides; hash joins build a table in memory. Broadcast joins skip the shuffle for tables under 10 MB by default (spark.sql.autoBroadcastJoinThreshold).
  • Aggregation and expression evaluation. Group-bys, window functions, casts, string and date functions, run row by row or by generated code on the JVM.
  • Spill. When a sort, join or aggregation does not fit its memory, Spark writes it to disk. The UI reports "shuffle spill (memory)" as the deserialized size and "shuffle spill (disk)" as the serialized size.
  • Serialization. Task closures, shuffle records and cached data are serialized. Spark defaults to Java serialization, which its tuning guide calls "often quite slow". Kryo is described there as "significantly faster and more compact than Java serialization (often as much as 10x)", but it needs class registration and is not the default.
  • JVM garbage collection. The task table in the UI carries a GC time column, the total JVM garbage collection time for the task. Spark sets aside 60% of the heap after a 300 MiB reserve for execution and storage by default (spark.memory.fraction 0.6); the rest holds user data structures and metadata.

What published measurements say about the split

The most cited measurement is the NSDI 2015 paper "Making Sense of Performance in Data Analytics Frameworks" from UC Berkeley, which instrumented Spark on two SQL benchmarks and one production workload. Its stated findings: CPU, not I/O, is often the bottleneck; improving network performance can improve job completion time by a median of at most 2%; and the causes of most stragglers can be identified. The paper dates from 2015, so it measured a much older Spark than the current release.

The Spark tuning guide, read 2026-11-10, says something different in its summary: because of the in-memory nature of most Spark computations, programs can be bottlenecked by CPU, network bandwidth or memory, and "most often, if the data fits in memory, the bottleneck is network bandwidth". Both statements are published; neither is a measurement of your job. A shuffle-heavy job on a small cluster with slow networking can look like the guide. A cached, compute-heavy job can look like the paper.

The Photon paper (SIGMOD 2022, Databricks) takes the CPU-bound reading: it describes replacing the JVM execution layer with a C++ vectorized engine, and reports customer queries averaging 3x faster than the previous Databricks Runtime, with a maximum above 10x. Databricks documents that queries shorter than about two seconds see no meaningful gain, which is a reminder that fixed costs such as planning and scheduling do not shrink with a faster operator.

Which option attacks which stage

Mapping from stage to the mechanism each project documents. Sources are listed at the end of the guide.
StageWhat changes it, per the project's own documentation
Scan and decodeNative Parquet readers in Apache DataFusion Comet and Apache Gluten with Velox; Databricks Photon; NVIDIA's cuDF plugin reads on the GPU
ShuffleColumnar shuffle managers (Apache DataFusion Comet, Apache Gluten); remote shuffle services Celeborn and Uniffle (Apache Auron, Apache Gluten); NVIDIA's RAPIDS Shuffle Manager
Sort, join, aggregationFPGA (DualBird says its F2 path covers sort, join and aggregation), GPU kernels (NVIDIA), Speedata's APU, native C++ or Rust operators
Spill and memoryNative engines use off-heap memory that the JVM does not manage, so they need off-heap settings; DualBird says disk spills effectively disappear on its path (DualBird's claim, not independently tested)
Serialization and GCNative engines keep data in columnar buffers outside the heap; Flarion says part of a job, including the driver and often the final write, still runs on the JVM heap

How to get your own split from the Spark UI

Turn on event logs before you run: spark.eventLog.enabled defaults to false. With the logs, the History Server rebuilds the same UI after the application ends. Then read three places.

On the Stages tab, sort stages by duration and look at the three longest. For each, compare shuffle read size, shuffle write size and the two spill figures against the input size. A stage that spills many times its input is memory-bound. On the task table, compare GC time with task duration, and scheduler delay and result serialization time with the median task. On the SQL tab, find the join type Spark chose and whether AQE changed it.

Record the split for your five most expensive jobs before any trial. The guide on running a two-week trial uses the same numbers as its baseline, and the guide on tuning shows which stage each setting touches.

Sources

  1. Making Sense of Performance in Data Analytics Frameworks, NSDI 2015, read 10 Nov 2026
  2. Apache Spark documentation: Tuning Spark, read 10 Nov 2026
  3. Apache Spark documentation: SQL performance tuning, read 10 Nov 2026
  4. Apache Spark documentation: Web UI, read 10 Nov 2026
  5. Photon: A Fast Query Engine for Lakehouse Systems, SIGMOD 2022, read 10 Nov 2026
  6. Databricks documentation: What is Photon, read 10 Nov 2026
  7. Flarion FAQs, read 10 Nov 2026

Last reviewed 10 Nov 2026.