The Ledger, read 10 Nov 2026
The Spark Accelerator Ledger: every published benchmark, with who ran it
Download the Ledger (CSV)How entries are recorded
All 21 entries, newest first
| Published | Option | Test | Result as published | Ran by | Method published |
|---|---|---|---|---|---|
| 2026-11-03 | Apache DataFusion Comet | Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-11-03 | Apache DataFusion Comet | Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-09 | Flarion | Flarion TPC-DS SF100, six engines on one machine | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-09 | Flarion | Flarion TPC-DS SF1000, six engines on one machine | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-08-25 | NVIDIA cuDF plugin for Apache Spark | GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) |
| 2026-06-11 | DualBird | DualBird sort-based Iceberg compaction, 100 GB | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found |
| 2026-06-11 | Speedata | Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched |
| 2025-08-21 | DataPelago | Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No |
| 2025-05-30 | Google Cloud Lightning Engine | Lightning Engine against open source Spark, TPC-H-like 10 TB | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No |
| 2022 | Databricks Photon | Photon customer speedups, SIGMOD 2022 paper | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No |
| 2021-11-02 | Databricks Photon | Databricks audited 100 TB TPC-DS result on Photon | 32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba) | Databricks, audited by the TPC (its maker) | No (a TPC audit report exists under TPC rules; not a repeatable open script) |
| Not stated | Amazon EMR runtime for Apache Spark | EMR runtime claim on the AWS features page | AWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performance | AWS (its maker) | Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results |
| Not stated | Apache Auron | Auron TPC-DS 1 TB against Apache Spark 3.5.6 | Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image only | Apache Auron project (its maker) | Yes, tools and full Spark config in the auron-project/tpcds-tools repository |
| Not stated | Apache Gluten with Velox | Gluten with Velox, TPC-DS 3 TB (project homepage) | Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queries | Apache Gluten project (its maker) | No |
| Not stated | Apache Gluten with Velox | Gluten with Velox, TPC-H 3 TB (project homepage) | Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queries | Apache Gluten project (its maker) | No |
| Not stated | DualBird | DualBird 110 GB enrichment pipeline (join denormalization) | DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not stated | DualBird (its maker) | No |
| Not stated | DualBird | DualBird single task performance tile | DualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark) | DualBird (its maker) | No |
| Not stated | Google Cloud Lightning Engine | Lightning Engine claim on Google's pricing page | Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name | Google Cloud (its maker) | No |
| Not stated | Google Cloud Lightning Engine | Native Query Execution on Axion ARM, Google's documentation | Google Cloud's wording: 2.5x to 4x speedup over open source Spark | Google Cloud (its maker) | No |
| Not stated | NVIDIA cuDF plugin for Apache Spark | NVIDIA comparison with open-source Spark on EC2, NDS 3 TB | NVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as published | NVIDIA (its maker) | No (the NDS automation is public; the comparison is described on the overview page only) |
| Not stated | Speedata | Speedata production Spark workload tile | Speedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing time | Speedata (its maker) | No |
Entries per option
| Option | Entries | Run by its maker | Run by another party |
|---|---|---|---|
| Amazon EMR runtime for Apache Spark | 1 | 1 | 0 |
| Apache Auron | 1 | 1 | 0 |
| Apache DataFusion Comet | 2 | 2 | 0 |
| Apache Gluten with Velox | 2 | 2 | 0 |
| Databricks Photon | 2 | 2 | 0 |
| DataPelago | 1 | 1 | 0 |
| DualBird | 3 | 3 | 0 |
| Flarion | 2 | 2 | 0 |
| Google Cloud Lightning Engine | 3 | 3 | 0 |
| Microsoft Fabric native execution engine | 0 | 0 | 0 |
| NVIDIA cuDF plugin for Apache Spark | 2 | 2 | 0 |
| Speedata | 2 | 1 | 1 |
Download and cite
CSV: /datasets/spark-accelerator-ledger.csv. JSON: /datasets/spark-accelerator-ledger.json. Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0).
Sources
- Comet TPC-DS benchmark results, read 10 Nov 2026
- Flarion benchmarks page, read 10 Nov 2026
- AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
- DualBird Iceberg compaction benchmark post, read 10 Nov 2026
- Speedata post on the SIGMOD paper, read 10 Nov 2026
- DataPelago Nucleus versus cuDF post, read 10 Nov 2026
- Google Lightning Engine launch blog, read 10 Nov 2026
- Photon paper, SIGMOD 2022, read 10 Nov 2026
- Databricks TPC-DS record announcement, read 10 Nov 2026
- Amazon EMR Spark features page, read 10 Nov 2026
- Apache Auron benchmark page, read 10 Nov 2026
- Apache Gluten homepage, read 10 Nov 2026
- DualBird home page, read 10 Nov 2026
- Google Managed Service for Apache Spark pricing, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- NVIDIA Spark plugin overview, read 10 Nov 2026
- Speedata home page, read 10 Nov 2026
Last reviewed 10 Nov 2026.