Skip to content
Spark Acceleration Compare

Situation, read 10 Nov 2026

What to check when nightly Spark pipelines no longer finish before morning

All situationsOpen the Ledger

5 things to check, with the published facts that answer each

Which runtimes have been published, on what workload and size?

Compare seconds and minutes only within one test. Each test below used its own cluster and baseline.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS says up to 4.5x faster than open-source equivalents, with no workload or scale stated.Amazon EMR Spark features page, 10 Nov 2026
Apache AuronThe Auron project says about 2x faster than Spark 3.5 on TPC-DS 1 TB (Spark 3.5.6 against Auron 6.0.0-preview on YARN); per-query numbers are a chart image only.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometTPC-DS derived, scale factor 1000, EKS on r6i: Spark 4.2.0 994 s, Comet 1.1.0 601 s, Comet 1.2.0 488 s.Comet TPC-DS benchmark results, 10 Nov 2026
Apache Gluten with VeloxThe Gluten homepage reports 3.34x on TPC-H and 3.02x on TPC-DS at 3 TB with Spark 3.3.1 on Intel hardware, with no method linked.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonThe SIGMOD 2022 paper reports an average customer speedup of 3x over the previous Databricks Runtime, with a maximum over 10x.Photon paper, SIGMOD 2022, 10 Nov 2026
DataPelagoDataPelago publishes operator microbenchmarks against NVIDIA cuDF (up to 10.5x on project, 8.2x on K-largest selection). An end-to-end Spark benchmark: none found on 2026-11-10.DataPelago Nucleus versus cuDF post, 10 Nov 2026
DualBirdDualBird's June 2026 post says about 12x to 20x faster Spark task execution on a 100 GB Iceberg compaction; its home page shows 6x to 18x on a 110 GB enrichment pipeline with no method.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionFlarion, TPC-DS SF1000 on one i4i.8xlarge: Spark 3.5.5 135.6 min, Flarion 38.3 min, Photon 30.4 min.Flarion benchmarks page, 10 Nov 2026
Google Cloud Lightning EngineGoogle reports 3.6x on TPC-H-like queries at 10 TB and, on its pricing page, up to 4.9x with no workload stated.Google Lightning Engine launch blog, 10 Nov 2026
Microsoft Fabric native execution engineA benchmark number for the Fabric native execution engine: none found on 2026-11-10.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkTPC-DS 3 TB on 8 nodes of EMR on EKS: GPU 281 s, CPU clusters 1,010 to 1,043 s (AWS and NVIDIA).AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata's home page shows 62.7x on production Spark workloads, with no customer, data size or method.Speedata home page, 10 Nov 2026

Which parts of your jobs stay on the old path?

An accelerator only helps the operators it takes over. Jobs heavy in UDFs or RDDs spend most of their time on the path that did not change.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache Gluten with VeloxGluten falls back through columnar-to-row conversions; the docs list ANSI mode as always falling back and Parquet as the only fully supported scan format.Gluten Velox backend limitations, 10 Nov 2026
Databricks PhotonDatabricks says Photon does not support UDFs, RDD APIs or Dataset APIs, and queries under about two seconds see no meaningful gain.Databricks Photon documentation, 10 Nov 2026
DataPelagoDataPelago tells users to avoid heavy use of RDDs and UDFs, and lists ANSI mode as running on the JVM.DataPelago Accelerator for Spark page, 10 Nov 2026
DualBirdA DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet.DualBird post, A Simpler Spark, 10 Nov 2026
FlarionFlarion says UDFs and UDAFs run on Spark as before, and the driver, planning, unsupported operators and often the final write still run on the JVM heap.Flarion FAQ, 10 Nov 2026
Google Cloud Lightning EngineGoogle says Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineMicrosoft says unsupported operators fall back to JVM Spark, and lists casting, timezone, round() and collect_list ordering differences on Runtime 1.3.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA says RDDs, GraphX and APIs that take Scala or Java functions are not accelerated, and unsupported operators fall back to the CPU.NVIDIA Spark plugin FAQ, 10 Nov 2026
SpeedataSpeedata says SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU.Speedata product page, 10 Nov 2026

Can you predict the gain before you change the cluster?

A pre-check on your own logs separates a TPC-DS result from your pipelines.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache DataFusion CometComet marks fallback reasons in plan explain output, which shows what stays on Spark.Comet compatibility guide, 10 Nov 2026
Databricks PhotonDatabricks tells users to check the Spark UI to see Photon fallback.Databricks Photon documentation, 10 Nov 2026
FlarionFlarion offers a free assessment that simulates each job from logs and reports projected runtime and compute.Flarion pricing page, 10 Nov 2026
Google Cloud Lightning EngineGoogle documents a qualification tool for checking jobs.Google Lightning Engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA documents a Qualification tool, a Profiling tool and an AutoTuner in its user guide.NVIDIA Spark plugin user guide, 10 Nov 2026
SpeedataSpeedata's Workload Analyzer predicts per-query acceleration before purchase from uploaded Spark logs.Speedata Workload Analyzer page, 10 Nov 2026

Is the slow part compute, or shuffle, spill and I/O?

Native engines speed up compute. A pipeline waiting on shuffle, small files or object storage may not move much.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron supports Celeborn 0.5 and 0.6 and Uniffle 0.10 as remote shuffle services.Apache Auron build script, 10 Nov 2026
Apache DataFusion CometComet supports remote shuffle through Celeborn.Comet Iceberg guide, 10 Nov 2026
Apache Gluten with VeloxGluten supports Celeborn 0.3 to 0.5 and Uniffle 0.9.2 for remote shuffle.Gluten Velox Iceberg page, 10 Nov 2026
Databricks PhotonDatabricks says queries under about two seconds see no meaningful gain from Photon.Databricks Photon documentation, 10 Nov 2026
DualBirdDualBird says tens of Spark settings become irrelevant, spills effectively disappear and skew stops causing spills. A later post says skew still costs time, but less. These are DualBird's claims.DualBird product page, 10 Nov 2026
Google Cloud Lightning EngineGoogle says I/O-bound or RDD-heavy jobs may suit the standard engine.Google Lightning Engine documentation, 10 Nov 2026

Can you rerun the benchmark yourself?

A method that can be repeated on your own data can be checked; a number that can only be read cannot.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS publishes aws-samples/emr-spark-benchmark for a 3 TB TPC-DS run on OSS Spark and EMR; the repository states no results.aws-samples emr-spark-benchmark repository, 10 Nov 2026
Apache AuronThe Auron benchmark page links its data generator and runner tools and the full Spark configuration.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometComet publishes benchmark scripts, a data generation repository and its non-default Spark settings.Comet benchmarking guide, 10 Nov 2026
Apache Gluten with VeloxGluten ships the gluten-it tool for TPC-style runs; the homepage numbers do not link a run configuration.Gluten integration test tool README, 10 Nov 2026
DualBirdDualBird's white paper has a landing page, and the full document is behind a form. No public repository was found.DualBird compaction white paper page, 10 Nov 2026
FlarionFlarion publishes its machine, builds, settings and caps, says the open-source engines can be rerun from public releases, and does not offer Flarion for download.Flarion benchmarks page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA publishes NDS benchmark automation in a public Apache-2.0 repository and notes NDS is not comparable to published TPC-DS results.NVIDIA NDS benchmark README, 10 Nov 2026
SpeedataSpeedata's customer tiles have no published method, and its white paper is behind a form.Speedata GitHub organization, 10 Nov 2026

Sources

  1. Amazon EMR Spark features page, read 10 Nov 2026
  2. Apache Auron benchmark page, read 10 Nov 2026
  3. Comet TPC-DS benchmark results, read 10 Nov 2026
  4. Apache Gluten homepage, read 10 Nov 2026
  5. Photon paper, SIGMOD 2022, read 10 Nov 2026
  6. DataPelago Nucleus versus cuDF post, read 10 Nov 2026
  7. DualBird Iceberg compaction benchmark post, read 10 Nov 2026
  8. Flarion benchmarks page, read 10 Nov 2026
  9. Google Lightning Engine launch blog, read 10 Nov 2026
  10. Fabric native execution engine documentation, read 10 Nov 2026
  11. AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
  12. Speedata home page, read 10 Nov 2026
  13. Gluten Velox backend limitations, read 10 Nov 2026
  14. Databricks Photon documentation, read 10 Nov 2026
  15. DataPelago Accelerator for Spark page, read 10 Nov 2026
  16. DualBird post, A Simpler Spark, read 10 Nov 2026
  17. Flarion FAQ, read 10 Nov 2026
  18. Google Lightning Engine documentation, read 10 Nov 2026
  19. NVIDIA Spark plugin FAQ, read 10 Nov 2026
  20. Speedata product page, read 10 Nov 2026
  21. Comet compatibility guide, read 10 Nov 2026
  22. Flarion pricing page, read 10 Nov 2026
  23. NVIDIA Spark plugin user guide, read 10 Nov 2026
  24. Speedata Workload Analyzer page, read 10 Nov 2026
  25. Apache Auron build script, read 10 Nov 2026
  26. Comet Iceberg guide, read 10 Nov 2026
  27. Gluten Velox Iceberg page, read 10 Nov 2026
  28. DualBird product page, read 10 Nov 2026
  29. aws-samples emr-spark-benchmark repository, read 10 Nov 2026
  30. Comet benchmarking guide, read 10 Nov 2026
  31. Gluten integration test tool README, read 10 Nov 2026
  32. DualBird compaction white paper page, read 10 Nov 2026
  33. NVIDIA NDS benchmark README, read 10 Nov 2026
  34. Speedata GitHub organization, read 10 Nov 2026

Last reviewed 10 Nov 2026.