Skip to content
Spark Acceleration Compare

The Ledger, read 10 Nov 2026

The Spark Accelerator Ledger: every published benchmark, with who ran it

Download the Ledger (CSV)How entries are recorded

All 21 entries, newest first

No number here is derived, averaged or estimated. Results from different entries ran on different hardware and data.
PublishedOptionTestResult as publishedRan byMethod published
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
2026-09FlarionFlarion TPC-DS SF100, six engines on one machineTotal query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 sFlarion (its maker)Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found
2026-09FlarionFlarion TPC-DS SF1000, six engines on one machineTotal query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 minFlarion (its maker)Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found
2026-08-25NVIDIA cuDF plugin for Apache SparkGPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TBGPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7xAWS with NVIDIA (its maker)Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked)
2026-06-11DualBirdDualBird sort-based Iceberg compaction, 100 GBAbout 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published)DualBird (its maker)Behind a sign-up form (white paper dated 2026-07-01); no public repository found
2026-06-11SpeedataSpeedata APU on TPC-H SF100, Microsoft CoddSpeed paperAPU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL ServerMicrosoft (paper); numbers read from Speedata's blog (another party)Yes, in the paper; the paper itself was not fetched
2025-08-21DataPelagoNucleus v3.0 against NVIDIA cuDF 25.04.00, operator testsUp to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDFDataPelago (its maker)No
2025-05-30Google Cloud Lightning EngineLightning Engine against open source Spark, TPC-H-like 10 TB3.6x acceleration over open source SparkGoogle Cloud (its maker)No
2022Databricks PhotonPhoton customer speedups, SIGMOD 2022 paperAverage speedup of 3x over the previous Databricks Runtime; maximum speedup over 10xDatabricks authors (its maker)No
2021-11-02Databricks PhotonDatabricks audited 100 TB TPC-DS result on Photon32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba)Databricks, audited by the TPC (its maker)No (a TPC audit report exists under TPC rules; not a repeatable open script)
Not statedAmazon EMR runtime for Apache SparkEMR runtime claim on the AWS features pageAWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performanceAWS (its maker)Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results
Not statedApache AuronAuron TPC-DS 1 TB against Apache Spark 3.5.6Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image onlyApache Auron project (its maker)Yes, tools and full Spark config in the auron-project/tpcds-tools repository
Not statedApache Gluten with VeloxGluten with Velox, TPC-DS 3 TB (project homepage)Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queriesApache Gluten project (its maker)No
Not statedApache Gluten with VeloxGluten with Velox, TPC-H 3 TB (project homepage)Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queriesApache Gluten project (its maker)No
Not statedDualBirdDualBird 110 GB enrichment pipeline (join denormalization)DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not statedDualBird (its maker)No
Not statedDualBirdDualBird single task performance tileDualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark)DualBird (its maker)No
Not statedGoogle Cloud Lightning EngineLightning Engine claim on Google's pricing pageGoogle Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not nameGoogle Cloud (its maker)No
Not statedGoogle Cloud Lightning EngineNative Query Execution on Axion ARM, Google's documentationGoogle Cloud's wording: 2.5x to 4x speedup over open source SparkGoogle Cloud (its maker)No
Not statedNVIDIA cuDF plugin for Apache SparkNVIDIA comparison with open-source Spark on EC2, NDS 3 TBNVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as publishedNVIDIA (its maker)No (the NDS automation is public; the comparison is described on the overview page only)
Not statedSpeedataSpeedata production Spark workload tileSpeedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing timeSpeedata (its maker)No

Entries per option

Download and cite

CSV: /datasets/spark-accelerator-ledger.csv. JSON: /datasets/spark-accelerator-ledger.json. Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0).

Sources

  1. Comet TPC-DS benchmark results, read 10 Nov 2026
  2. Flarion benchmarks page, read 10 Nov 2026
  3. AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
  4. DualBird Iceberg compaction benchmark post, read 10 Nov 2026
  5. Speedata post on the SIGMOD paper, read 10 Nov 2026
  6. DataPelago Nucleus versus cuDF post, read 10 Nov 2026
  7. Google Lightning Engine launch blog, read 10 Nov 2026
  8. Photon paper, SIGMOD 2022, read 10 Nov 2026
  9. Databricks TPC-DS record announcement, read 10 Nov 2026
  10. Amazon EMR Spark features page, read 10 Nov 2026
  11. Apache Auron benchmark page, read 10 Nov 2026
  12. Apache Gluten homepage, read 10 Nov 2026
  13. DualBird home page, read 10 Nov 2026
  14. Google Managed Service for Apache Spark pricing, read 10 Nov 2026
  15. Google Lightning Engine documentation, read 10 Nov 2026
  16. NVIDIA Spark plugin overview, read 10 Nov 2026
  17. Speedata home page, read 10 Nov 2026

Last reviewed 10 Nov 2026.