# The Spark Accelerator Ledger: every published benchmark, with who ran it

The Ledger, read 10 Nov 2026

**The short answer.** The Spark Accelerator Ledger lists 21 published benchmarks for 12 ways to speed up Spark, newest first. 20 were run by the maker of the option tested and 1 by another named party. Each entry gives the workload, data size, hardware, the result as published and whether the method is public.

## All 21 entries, newest first

| Published | Option | Test | Result as published | Ran by | Method published |
| --- | --- | --- | --- | --- | --- |
| 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-08-25 | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) |
| 2026-06-11 | [DualBird](/options/dualbird/) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found |
| 2026-06-11 | [Speedata](/options/speedata/) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched |
| 2025-08-21 | [DataPelago](/options/datapelago/) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No |
| 2025-05-30 | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No |
| 2022 | [Databricks Photon](/options/databricks-photon/) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No |
| 2021-11-02 | [Databricks Photon](/options/databricks-photon/) | [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) | 32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba) | Databricks, audited by the TPC (its maker) | No (a TPC audit report exists under TPC rules; not a repeatable open script) |
| Not stated | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) | AWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performance | AWS (its maker) | Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results |
| Not stated | [Apache Auron](/options/apache-auron/) | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) | Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image only | Apache Auron project (its maker) | Yes, tools and full Spark config in the auron-project/tpcds-tools repository |
| Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) | Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queries | Apache Gluten project (its maker) | No |
| Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) | Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queries | Apache Gluten project (its maker) | No |
| Not stated | [DualBird](/options/dualbird/) | [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) | DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not stated | DualBird (its maker) | No |
| Not stated | [DualBird](/options/dualbird/) | [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) | DualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark) | DualBird (its maker) | No |
| Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) | Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name | Google Cloud (its maker) | No |
| Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) | Google Cloud's wording: 2.5x to 4x speedup over open source Spark | Google Cloud (its maker) | No |
| Not stated | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) | NVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as published | NVIDIA (its maker) | No (the NDS automation is public; the comparison is described on the overview page only) |
| Not stated | [Speedata](/options/speedata/) | [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) | Speedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing time | Speedata (its maker) | No |

No number here is derived, averaged or estimated. Results from different entries ran on different hardware and data.

## Entries per option

| Option | Entries | Run by its maker | Run by another party |
| --- | --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | 1 | 1 | 0 |
| [Apache Auron](/options/apache-auron/) | 1 | 1 | 0 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | 2 | 2 | 0 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | 2 | 2 | 0 |
| [Databricks Photon](/options/databricks-photon/) | 2 | 2 | 0 |
| [DataPelago](/options/datapelago/) | 1 | 1 | 0 |
| [DualBird](/options/dualbird/) | 3 | 3 | 0 |
| [Flarion](/options/flarion/) | 2 | 2 | 0 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | 3 | 3 | 0 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | 0 | 0 | 0 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | 2 | 2 | 0 |
| [Speedata](/options/speedata/) | 2 | 1 | 1 |

Alphabetical by option.

## Download and cite

CSV: [/datasets/spark-accelerator-ledger.csv](/datasets/spark-accelerator-ledger.csv). JSON: [/datasets/spark-accelerator-ledger.json](/datasets/spark-accelerator-ledger.json). Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0).

## Sources

- Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026)
- AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026)
- DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026)
- Speedata post on the SIGMOD paper: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026)
- DataPelago Nucleus versus cuDF post: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026)
- Google Lightning Engine launch blog: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026)
- Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026)
- Databricks TPC-DS record announcement: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026)
- Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026)
- Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026)
- Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026)
- DualBird home page: https://dualbird.io/ (read 10 Nov 2026)
- Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)
- Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)
- NVIDIA Spark plugin overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026)
- Speedata home page: https://www.speedata.io/ (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/
