# Spark Acceleration Compare > Twelve ways to speed up Spark without rewriting jobs, with a Ledger of published benchmarks and a Change Budget of what each makes a team change. Every page has a markdown twin at its address plus index.md (or send Accept: text/markdown). Every fact carries its source and the date it was read (10 Nov 2026). Datasets: https://sparkaccelerationcompare.com/datasets/ (CSV and JSON, CC BY 4.0). ## Home - [Ways to speed up Spark without rewriting your jobs, compared](https://sparkaccelerationcompare.com/index.md): Twelve ways to speed up Spark without rewriting jobs, with a Ledger of published benchmarks and a Change Budget of what each makes a team change. ## Index - [The options, A to Z](https://sparkaccelerationcompare.com/options/index.md): 12 ways to speed up Spark without rewriting jobs, each with the same fields. - [Any two options side by side](https://sparkaccelerationcompare.com/compare/index.md): 66 side-by-side pages: two options, the same rows, every cell sourced. - [Guides](https://sparkaccelerationcompare.com/guides/index.md): 10 guides for the engineer who owns the Spark bill, each with its sources. - [Glossary](https://sparkaccelerationcompare.com/glossary/index.md): 35 terms, each with a sourced example. - [News](https://sparkaccelerationcompare.com/news/index.md): 15 dated items from primary sources, newest first. - [Situations](https://sparkaccelerationcompare.com/situations/index.md): Five buyer situations, what to check in each and which published facts answer it. ## Profile - [Amazon EMR runtime for Apache Spark: AWS-tuned Spark runtime on EMR](https://sparkaccelerationcompare.com/options/emr-spark-runtime/index.md): Spark runtime from AWS: 100% API compatible per AWS, on EMR on EC2, EKS and Serverless, with a public TPC-DS harness. - [Apache Auron: Open-source Rust engine plugin for Spark](https://sparkaccelerationcompare.com/options/apache-auron/index.md): Apache-2.0 Spark plugin with a Rust native engine, a TPC-DS 1 TB method and builds for Spark 3.0 to 4.2. - [Apache DataFusion Comet: Open-source Spark plugin on DataFusion](https://sparkaccelerationcompare.com/options/datafusion-comet/index.md): Apache-2.0 Spark plugin on DataFusion with published TPC-DS scripts, settings and monthly releases since 1.0.0. - [Apache Gluten with Velox: Open-source Spark plugin offloading to Velox](https://sparkaccelerationcompare.com/options/apache-gluten-velox/index.md): Apache-2.0 Spark plugin offloading to Velox, with a built-in benchmark tool and a self-reported adopter list. - [Databricks Photon: Databricks's C++ vectorized engine](https://sparkaccelerationcompare.com/options/databricks-photon/index.md): Databricks's own C++ engine, on by default in classic jobs compute and documented only inside Databricks. - [DataPelago: Gluten-based CPU and GPU accelerator](https://sparkaccelerationcompare.com/options/datapelago/index.md): Proprietary Gluten-based Spark plugin for EKS, Dataproc and GKE, billed through cloud marketplaces. - [DualBird: FPGA Spark plugin on AWS F2](https://sparkaccelerationcompare.com/options/dualbird/index.md): FPGA-based Spark plugin for AWS F2 instances, with one published worked example, an Iceberg compaction test. - [Flarion: Rust and DataFusion engine as one JAR](https://sparkaccelerationcompare.com/options/flarion/index.md): Proprietary native Spark engine as one JAR, with published per-core list prices and a six-engine TPC-DS page. - [Google Cloud Lightning Engine: Premium Spark engine on Google Cloud](https://sparkaccelerationcompare.com/options/dataproc-lightning-engine/index.md): Google Cloud's Spark engine tier with optional native execution, priced per DCU-hour in us-central1. - [Microsoft Fabric native execution engine: Velox-based engine inside Fabric Spark](https://sparkaccelerationcompare.com/options/fabric-native-execution-engine/index.md): Velox and Gluten engine inside Fabric Spark on Azure, switched on with one toggle or one setting. - [NVIDIA cuDF plugin for Apache Spark: GPU plugin, formerly RAPIDS Accelerator](https://sparkaccelerationcompare.com/options/nvidia-rapids-spark/index.md): GPU plugin for Spark SQL and DataFrame jobs, with a cloud-provider benchmark that publishes runtime and cost per run. - [Speedata: PCIe analytics chip plus Dash Spark plugin](https://sparkaccelerationcompare.com/options/speedata/index.md): Custom-silicon Spark offload on a PCIe card, sold as servers, OEM hardware or a hosted AWS service. ## Compare - [Amazon EMR runtime for Apache Spark and Apache Auron side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and Apache Auron: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Apache DataFusion Comet side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and Apache DataFusion Comet: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Apache Gluten with Velox side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and Apache Gluten with Velox: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Databricks Photon side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and Databricks Photon: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and DataPelago side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and DataPelago: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and DualBird side by side](https://sparkaccelerationcompare.com/compare/dualbird-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Flarion side by side](https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-flarion/index.md): Amazon EMR runtime for Apache Spark and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-emr-spark-runtime/index.md): Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-fabric-native-execution-engine/index.md): Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-nvidia-rapids-spark/index.md): Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Amazon EMR runtime for Apache Spark and Speedata side by side](https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-speedata/index.md): Amazon EMR runtime for Apache Spark and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Apache DataFusion Comet side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-datafusion-comet/index.md): Apache Auron and Apache DataFusion Comet: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Apache Gluten with Velox side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-apache-gluten-velox/index.md): Apache Auron and Apache Gluten with Velox: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Databricks Photon side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-databricks-photon/index.md): Apache Auron and Databricks Photon: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and DataPelago side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-datapelago/index.md): Apache Auron and DataPelago: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and DualBird side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-dualbird/index.md): Apache Auron and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Flarion side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-flarion/index.md): Apache Auron and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-dataproc-lightning-engine/index.md): Apache Auron and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-fabric-native-execution-engine/index.md): Apache Auron and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-nvidia-rapids-spark/index.md): Apache Auron and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Auron and Speedata side by side](https://sparkaccelerationcompare.com/compare/apache-auron-vs-speedata/index.md): Apache Auron and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Apache Gluten with Velox side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-datafusion-comet/index.md): Apache DataFusion Comet and Apache Gluten with Velox: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Databricks Photon side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-datafusion-comet/index.md): Apache DataFusion Comet and Databricks Photon: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and DataPelago side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-datapelago/index.md): Apache DataFusion Comet and DataPelago: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and DualBird side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-dualbird/index.md): Apache DataFusion Comet and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Flarion side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-flarion/index.md): Apache DataFusion Comet and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-dataproc-lightning-engine/index.md): Apache DataFusion Comet and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-fabric-native-execution-engine/index.md): Apache DataFusion Comet and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-nvidia-rapids-spark/index.md): Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache DataFusion Comet and Speedata side by side](https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-speedata/index.md): Apache DataFusion Comet and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and Databricks Photon side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-databricks-photon/index.md): Apache Gluten with Velox and Databricks Photon: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and DataPelago side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-datapelago/index.md): Apache Gluten with Velox and DataPelago: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and DualBird side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-dualbird/index.md): Apache Gluten with Velox and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and Flarion side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-flarion/index.md): Apache Gluten with Velox and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-dataproc-lightning-engine/index.md): Apache Gluten with Velox and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-fabric-native-execution-engine/index.md): Apache Gluten with Velox and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-nvidia-rapids-spark/index.md): Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Apache Gluten with Velox and Speedata side by side](https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-speedata/index.md): Apache Gluten with Velox and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and DataPelago side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-datapelago/index.md): Databricks Photon and DataPelago: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and DualBird side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-dualbird/index.md): Databricks Photon and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and Flarion side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-flarion/index.md): Databricks Photon and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-dataproc-lightning-engine/index.md): Databricks Photon and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-fabric-native-execution-engine/index.md): Databricks Photon and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-nvidia-rapids-spark/index.md): Databricks Photon and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Databricks Photon and Speedata side by side](https://sparkaccelerationcompare.com/compare/databricks-photon-vs-speedata/index.md): Databricks Photon and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and DualBird side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-dualbird/index.md): DataPelago and DualBird: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and Flarion side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-flarion/index.md): DataPelago and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-dataproc-lightning-engine/index.md): DataPelago and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-fabric-native-execution-engine/index.md): DataPelago and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-nvidia-rapids-spark/index.md): DataPelago and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [DataPelago and Speedata side by side](https://sparkaccelerationcompare.com/compare/datapelago-vs-speedata/index.md): DataPelago and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [DualBird and Flarion side by side](https://sparkaccelerationcompare.com/compare/dualbird-vs-flarion/index.md): DualBird and Flarion: published results, code and cluster changes, where each runs, in the same rows with sources. - [DualBird and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-dualbird/index.md): DualBird and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [DualBird and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/dualbird-vs-fabric-native-execution-engine/index.md): DualBird and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [DualBird and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/dualbird-vs-nvidia-rapids-spark/index.md): DualBird and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [DualBird and Speedata side by side](https://sparkaccelerationcompare.com/compare/dualbird-vs-speedata/index.md): DualBird and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Flarion and Google Cloud Lightning Engine side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-flarion/index.md): Flarion and Google Cloud Lightning Engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Flarion and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-flarion/index.md): Flarion and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Flarion and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/flarion-vs-nvidia-rapids-spark/index.md): Flarion and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Flarion and Speedata side by side](https://sparkaccelerationcompare.com/compare/flarion-vs-speedata/index.md): Flarion and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Google Cloud Lightning Engine and Microsoft Fabric native execution engine side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/index.md): Google Cloud Lightning Engine and Microsoft Fabric native execution engine: published results, code and cluster changes, where each runs, in the same rows with sources. - [Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/index.md): Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Google Cloud Lightning Engine and Speedata side by side](https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-speedata/index.md): Google Cloud Lightning Engine and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark side by side](https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/index.md): Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark: published results, code and cluster changes, where each runs, in the same rows with sources. - [Microsoft Fabric native execution engine and Speedata side by side](https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-speedata/index.md): Microsoft Fabric native execution engine and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. - [NVIDIA cuDF plugin for Apache Spark and Speedata side by side](https://sparkaccelerationcompare.com/compare/nvidia-rapids-spark-vs-speedata/index.md): NVIDIA cuDF plugin for Apache Spark and Speedata: published results, code and cluster changes, where each runs, in the same rows with sources. ## Asset - [Spark Accelerator Ledger](https://sparkaccelerationcompare.com/ledger/index.md): 21 published Spark acceleration benchmarks with who ran each, on what and the result as published. - [The Change Budget](https://sparkaccelerationcompare.com/change-budget/index.md): What each of 12 ways to speed up Spark makes a team change: code, cluster, cloud and vendor. ## Method - [How the Ledger is recorded](https://sparkaccelerationcompare.com/ledger/method/index.md): What counts as a published benchmark, how entries are dated, labelled and ordered. - [How the Change Budget is built](https://sparkaccelerationcompare.com/change-budget/method/index.md): The four columns of the Change Budget and how each cell is sourced. - [How this site works](https://sparkaccelerationcompare.com/method/index.md): Spark Acceleration Compare lists the twelve ways to speed up Spark that it covers and does not rank them. ## Ledger - [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](https://sparkaccelerationcompare.com/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/index.md): Apache DataFusion Comet: TPC-DS derived queries, 103 queries, Parquet (zstd) in S3. Run by Apache DataFusion Comet project, published 2026-11-03. - [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](https://sparkaccelerationcompare.com/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/index.md): Apache DataFusion Comet: TPC-DS derived queries, 103 queries, Parquet (zstd) in S3, each query run twice, mean reported. Run by Apache DataFusion Comet project, publ - [Flarion TPC-DS SF100, six engines on one machine](https://sparkaccelerationcompare.com/ledger/flarion-tpcds-sf100-six-engines/index.md): Flarion: TPC-DS, 103 queries, sum of the lowest per-query times, two runs. Run by Flarion, published 2026-09. - [Flarion TPC-DS SF1000, six engines on one machine](https://sparkaccelerationcompare.com/ledger/flarion-tpcds-sf1000-six-engines/index.md): Flarion: TPC-DS, 103 queries, sum of per-query times, one run, 1,800 s cap per query. Run by Flarion, published 2026-09. - [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](https://sparkaccelerationcompare.com/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/index.md): NVIDIA cuDF plugin for Apache Spark: TPC-DS, all 103 queries, power run, Parquet on S3, median of 5 iterations. Run by AWS with NVIDIA, published 2026-08-25. - [DualBird sort-based Iceberg compaction, 100 GB](https://sparkaccelerationcompare.com/ledger/dualbird-iceberg-compaction-100gb/index.md): DualBird: Sort-based Apache Iceberg compaction on a synthetic denormalized event table (about 370 million rows, 64 columns, ZSTD level 3). Run by DualBird, published - [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](https://sparkaccelerationcompare.com/ledger/speedata-apu-tpch-sf100-microsoft-paper/index.md): Speedata: TPC-H queries Q07, Q10, Q12, Q13 and Q16 through Microsoft's coprocessor abstraction layer, standalone APU against single-core SQL Server. Run by Microsoft - [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](https://sparkaccelerationcompare.com/ledger/datapelago-nucleus-vs-cudf-operators/index.md): DataPelago: Project, filter, aggregate, hash aggregate, hash join and K-largest selection operators on a single GPU. Run by DataPelago, published 2025-08-21. - [Lightning Engine against open source Spark, TPC-H-like 10 TB](https://sparkaccelerationcompare.com/ledger/google-lightning-engine-tpch-like-10tb/index.md): Google Cloud Lightning Engine: TPC-H derived queries (not compliant with the TPC-H specification). Run by Google Cloud, published 2025-05-30. - [Photon customer speedups, SIGMOD 2022 paper](https://sparkaccelerationcompare.com/ledger/databricks-photon-customer-speedups-sigmod-2022/index.md): Databricks Photon: Customer production queries. Run by Databricks authors, published 2022. - [Databricks audited 100 TB TPC-DS result on Photon](https://sparkaccelerationcompare.com/ledger/databricks-photon-tpcds-100tb-audited/index.md): Databricks Photon: Official TPC-DS run on Databricks SQL. Run by Databricks, audited by the TPC, published 2021-11-02. - [EMR runtime claim on the AWS features page](https://sparkaccelerationcompare.com/ledger/amazon-emr-runtime-features-page-claim/index.md): Amazon EMR runtime for Apache Spark: Not stated on the page (Iceberg write also cited). Run by AWS, published Not stated. - [Auron TPC-DS 1 TB against Apache Spark 3.5.6](https://sparkaccelerationcompare.com/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/index.md): Apache Auron: TPC-DS, Parquet with zstd, run with a spark-sql-perf derived runner. Run by Apache Auron project, published Not stated. - [Gluten with Velox, TPC-DS 3 TB (project homepage)](https://sparkaccelerationcompare.com/ledger/apache-gluten-velox-tpcds-3tb/index.md): Apache Gluten with Velox: TPC-DS. Run by Apache Gluten project, published Not stated. - [Gluten with Velox, TPC-H 3 TB (project homepage)](https://sparkaccelerationcompare.com/ledger/apache-gluten-velox-tpch-3tb/index.md): Apache Gluten with Velox: TPC-H. Run by Apache Gluten project, published Not stated. - [DualBird 110 GB enrichment pipeline (join denormalization)](https://sparkaccelerationcompare.com/ledger/dualbird-enrichment-pipeline-110gb/index.md): DualBird: Enrichment pipeline with join denormalization. Run by DualBird, published Not stated. - [DualBird single task performance tile](https://sparkaccelerationcompare.com/ledger/dualbird-single-task-speedup/index.md): DualBird: Single Spark task at the same cluster size (home page tile; appears to be the compaction workload). Run by DualBird, published Not stated. - [Lightning Engine claim on Google's pricing page](https://sparkaccelerationcompare.com/ledger/google-lightning-engine-pricing-page-claim/index.md): Google Cloud Lightning Engine: Not stated. Run by Google Cloud, published Not stated. - [Native Query Execution on Axion ARM, Google's documentation](https://sparkaccelerationcompare.com/ledger/google-native-query-execution-axion/index.md): Google Cloud Lightning Engine: Analytical workloads (not specified). Run by Google Cloud, published Not stated. - [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](https://sparkaccelerationcompare.com/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/index.md): NVIDIA cuDF plugin for Apache Spark: NDS (TPC-DS derived), 103 queries. Run by NVIDIA, published Not stated. - [Speedata production Spark workload tile](https://sparkaccelerationcompare.com/ledger/speedata-production-spark-consolidation/index.md): Speedata: Enterprise AI data preprocessing on Spark; customer not named. Run by Speedata, published Not stated. ## About - [About Spark Acceleration Compare](https://sparkaccelerationcompare.com/about/index.md): Spark Acceleration Compare is a reference publication on ways to make Apache Spark faster without rewriting jobs. ## Policy - [Editorial policy](https://sparkaccelerationcompare.com/editorial-policy/index.md): Every fact on this site comes from a page that was fetched and read on a stated date, and each fact links to it. - [Corrections](https://sparkaccelerationcompare.com/corrections/index.md): To report a wrong or outdated fact, write to editor@sparkaccelerationcompare.com with the page, the sentence and a link to the source that shows the correct fact. - [Contact](https://sparkaccelerationcompare.com/contact/index.md): Write to editor@sparkaccelerationcompare.com for corrections, questions about a source, or a request for a missing option. - [Privacy](https://sparkaccelerationcompare.com/privacy/index.md): Spark Acceleration Compare uses Google Analytics 4, which sets cookies, to count visits. - [Terms of use](https://sparkaccelerationcompare.com/terms/index.md): The pages on Spark Acceleration Compare are provided for reference. ## Guide - [Where a Spark job spends its time: scan to JVM](https://sparkaccelerationcompare.com/guides/where-spark-time-goes/index.md): Where Spark time goes: scan, shuffle, sort, join, spill, serialization and the JVM, with figures from the Berkeley NSDI 2015 study and the Apache Spark documentation. - [What "no code change" means for Spark acceleration](https://sparkaccelerationcompare.com/guides/what-no-code-change-means/index.md): What no code change means for Spark acceleration: SQL and DataFrame versus RDD and UDF code, fallback, config flags and instance changes, per option. - [How to read a vendor Spark benchmark: seven checks](https://sparkaccelerationcompare.com/guides/reading-a-vendor-benchmark/index.md): Seven checks for a Spark acceleration benchmark: who ran it, workload, scale factor, hardware, baseline Spark version, cost basis and published method. - [GPU, FPGA and custom silicon for Spark in plain words](https://sparkaccelerationcompare.com/guides/hardware-acceleration-plain-words/index.md): What FPGA, GPU and custom-silicon accelerators do for Apache Spark, with DualBird on AWS F2, NVIDIA cuDF and Speedata C200 as examples, plus AWS facts. - [Spark performance tuning: what to do when it is not enough](https://sparkaccelerationcompare.com/guides/when-tuning-is-cheaper/index.md): Spark performance tuning checklist (AQE, shuffle partitions, broadcast joins, file sizes, skew) and how to tell when tuning is not enough. - [Native Spark engines: Auron, DataFusion, Photon, Velox](https://sparkaccelerationcompare.com/guides/native-engines-explained/index.md): What vectorized native Spark engines do (Auron, DataFusion, Photon, Velox): columnar execution, operator coverage, fallback, Spark versions and where each runs. - [GPU vs FPGA for Spark: the published facts on each](https://sparkaccelerationcompare.com/guides/gpu-vs-fpga-for-spark/index.md): GPU versus FPGA for Apache Spark: what DualBird and NVIDIA have each published about speed, cost, hardware, operators and availability, side by side with sources. - [Spark acceleration options on Amazon EMR and EKS](https://sparkaccelerationcompare.com/guides/spark-acceleration-on-emr-and-eks/index.md): Which Spark accelerators run on Amazon EMR and EKS and what each needs: instance families, plugins, EMR release labels, Spark versions and billing. - [Operator coverage and fallback: checking what an accelerator covers](https://sparkaccelerationcompare.com/guides/operator-coverage-and-fallback/index.md): How to check which Spark operators an accelerator covers and what happens on fallback, using each project's supported-operator lists, explain output and the Spark UI. - [How to run a fair two-week Spark accelerator trial](https://sparkaccelerationcompare.com/guides/trialing-an-accelerator/index.md): A two-week plan to trial a Spark accelerator on your own jobs: baseline, same data, same cluster cost, cost per job, event logs and the Spark UI, with sources. ## Glossary - [Adaptive Query Execution: definition and example](https://sparkaccelerationcompare.com/glossary/adaptive-query-execution/index.md): Adaptive Query Execution (AQE) lets Spark SQL re-optimize a running query from runtime statistics. - [Broadcast hash join: definition and example](https://sparkaccelerationcompare.com/glossary/broadcast-hash-join/index.md): A broadcast hash join copies a small table to every executor and joins it against the large table in place, so the large side is never shuffled. - [Sort-merge join: definition and example](https://sparkaccelerationcompare.com/glossary/sort-merge-join/index.md): A sort-merge join sorts both sides of a join on the join key and then walks the two sorted streams together. - [Shuffle: definition and example](https://sparkaccelerationcompare.com/glossary/shuffle/index.md): A shuffle is Spark's mechanism for redistributing data so that it is grouped differently across partitions. - [Spill: definition and example](https://sparkaccelerationcompare.com/glossary/spill/index.md): Spill is what happens when Spark cannot hold shuffle or aggregation data in execution memory and writes it to local disk instead. - [Whole-stage code generation: definition and example](https://sparkaccelerationcompare.com/glossary/whole-stage-code-generation/index.md): Whole-stage code generation fuses a chain of Spark SQL operators into one generated Java function, so rows pass between operators in CPU registers instead of through - [Tungsten: definition and example](https://sparkaccelerationcompare.com/glossary/tungsten/index.md): Tungsten is the Apache Spark project, announced by Databricks on April 28, 2015, to bring Spark closer to bare metal, so hardware rather than JVM overhead sets the l - [Catalyst: definition and example](https://sparkaccelerationcompare.com/glossary/catalyst/index.md): Catalyst is the query optimizer inside Spark SQL. - [Vectorized execution: definition and example](https://sparkaccelerationcompare.com/glossary/vectorized-execution/index.md): Vectorized execution processes a batch of values from one column per function call instead of one row per call. - [Columnar batch: definition and example](https://sparkaccelerationcompare.com/glossary/columnar-batch/index.md): A columnar batch is a group of rows stored column by column in memory, so a filter reads one contiguous array per column. - [Apache Arrow: definition and example](https://sparkaccelerationcompare.com/glossary/apache-arrow/index.md): Apache Arrow defines a language-independent columnar memory format for flat and nested data, laid out for analytic work on CPUs and GPUs. - [Velox: definition and example](https://sparkaccelerationcompare.com/glossary/velox/index.md): Velox is an open source C++ library of execution components, created by Meta, that takes an optimized query plan as input and performs the computation. - [DataFusion: definition and example](https://sparkaccelerationcompare.com/glossary/datafusion/index.md): Apache DataFusion is a query engine written in Rust that uses the Apache Arrow in-memory format. - [Substrait: definition and example](https://sparkaccelerationcompare.com/glossary/substrait/index.md): Substrait is a cross-language specification for describing compute operations on structured data, with serialized forms so that different systems can pass a query pl - [Off-heap memory: definition and example](https://sparkaccelerationcompare.com/glossary/off-heap-memory/index.md): Off-heap memory is memory Spark or a plugin allocates outside the JVM heap. - [Operator fallback: definition and example](https://sparkaccelerationcompare.com/glossary/operator-fallback/index.md): Operator fallback is what an accelerator does when it cannot run a Spark operator or expression: that part of the plan runs on regular Spark, and the rest runs on th - [Spark plugin (spark.plugins): definition and example](https://sparkaccelerationcompare.com/glossary/spark-plugin/index.md): A Spark plugin is a class loaded into the driver and executors through the spark.plugins setting. - [Executor: definition and example](https://sparkaccelerationcompare.com/glossary/executor/index.md): An executor is a process launched for a Spark application on a worker node. - [Partition: definition and example](https://sparkaccelerationcompare.com/glossary/partition/index.md): A partition is a slice of a dataset that one Spark task processes. - [Skew: definition and example](https://sparkaccelerationcompare.com/glossary/skew/index.md): Skew is an uneven spread of data across partitions, so a few tasks do most of the work while others finish early. - [Speculative execution: definition and example](https://sparkaccelerationcompare.com/glossary/speculative-execution/index.md): Speculative execution relaunches a slow task on another executor and keeps whichever copy finishes first. - [TPC-DS: definition and example](https://sparkaccelerationcompare.com/glossary/tpc-ds/index.md): TPC-DS is a decision-support benchmark from the Transaction Processing Performance Council, for ad hoc analytic queries over a retail-style data model. - [TPC-H: definition and example](https://sparkaccelerationcompare.com/glossary/tpc-h/index.md): TPC-H is a decision-support benchmark of business-oriented ad hoc queries and concurrent data modifications. - [Scale factor: definition and example](https://sparkaccelerationcompare.com/glossary/scale-factor/index.md): Scale factor is the knob that sets how much data a TPC-style benchmark generates. - [FPGA: definition and example](https://sparkaccelerationcompare.com/glossary/fpga/index.md): An FPGA, or field-programmable gate array, is a chip whose logic can be reconfigured after manufacture to run one workload as a hardware pipeline. - [GPU: definition and example](https://sparkaccelerationcompare.com/glossary/gpu/index.md): A GPU is a processor with thousands of small cores built for data-parallel work. - [APU (Analytics Processing Unit): definition and example](https://sparkaccelerationcompare.com/glossary/apu/index.md): An APU in this context is Speedata's Analytics Processing Unit, a chip that runs Spark SQL operations in silicon rather than in memory. - [AWS F2 instance: definition and example](https://sparkaccelerationcompare.com/glossary/aws-f2-instance/index.md): AWS F2 instances are second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB of high-bandwidth memory each, and 3rd-generation A - [EMR release label: definition and example](https://sparkaccelerationcompare.com/glossary/emr-release-label/index.md): An EMR release label such as emr-7.14.0 names a fixed bundle of open source applications, each at a stated version, that Amazon EMR installs. - [EMR on EKS: definition and example](https://sparkaccelerationcompare.com/glossary/emr-on-eks/index.md): Amazon EMR on EKS is a deployment option that runs open source big data frameworks such as Spark on Amazon Elastic Kubernetes Service. - [Spark Operator: definition and example](https://sparkaccelerationcompare.com/glossary/spark-operator/index.md): The Kubernetes operator for Apache Spark, called the Spark Operator, lets you submit and manage Spark applications as Kubernetes custom resources. - [Iceberg compaction (rewrite_data_files): definition and example](https://sparkaccelerationcompare.com/glossary/iceberg-compaction/index.md): Iceberg compaction rewrites many small data files into fewer large ones. - [Parquet row group: definition and example](https://sparkaccelerationcompare.com/glossary/parquet-row-group/index.md): A Parquet row group is a horizontal slice of rows inside a Parquet file. - [Dynamic partition pruning: definition and example](https://sparkaccelerationcompare.com/glossary/dynamic-partition-pruning/index.md): Dynamic partition pruning skips partitions of a large table at runtime using the filter results of a joined smaller table. - [Cost per job: definition and example](https://sparkaccelerationcompare.com/glossary/cost-per-job/index.md): Cost per job is the total bill for one run of a Spark job: compute at the instance or per-vCPU rate times elapsed time, plus any platform uplift, storage and license ## News - [Apache Spark 4.3.0 released with stateful Real-Time Mode and SCD Type 2](https://sparkaccelerationcompare.com/news/spark-4-3-0-released/index.md): Apache Spark 4.3.0 was announced on October 30, 2026, the fourth release of the 4.x line. - [Apache Spark 4.2.1, 4.1.4 and 4.0.5 maintenance releases announced](https://sparkaccelerationcompare.com/news/spark-maintenance-releases-oct-2026/index.md): Apache Spark announced three maintenance releases on October 27, 2026: 4.2.1, 4.1.4 and 4.0.5. - [Apache Gluten 1.7.0 released with Spark 4.x test coverage and Delta 4 native write](https://sparkaccelerationcompare.com/news/gluten-1-7-0-released/index.md): Apache Gluten published release 1.7.0 on August 26, 2026, after three release candidates dated August 10, 17 and 21. - [Apache DataFusion Comet 1.1.0 released: 416 commits from 41 contributors](https://sparkaccelerationcompare.com/news/comet-1-1-0-released/index.md): Apache DataFusion Comet 1.1.0 was published on October 6, 2026. - [Apache DataFusion Comet 1.2.0 released: 399 commits from 40 contributors](https://sparkaccelerationcompare.com/news/comet-1-2-0-released/index.md): Apache DataFusion Comet 1.2.0 was published on November 3, 2026, four weeks after 1.1.0. - [NVIDIA cuDF plugin for Apache Spark 26.10.0 released, adds Spark 4.2.0 support](https://sparkaccelerationcompare.com/news/nvidia-cudf-plugin-26-10-0/index.md): NVIDIA published release v26.10.0 of its cuDF plugin for Apache Spark on November 5, 2026. - [Amazon EMR introduces Long Term Support, starting with emr-spark-8.1.0 on Spark 4.1](https://sparkaccelerationcompare.com/news/emr-spark-8-1-0-lts/index.md): Amazon EMR announced Long Term Support releases on September 22, 2026, starting with emr-spark-8.1.0 and Apache Spark 4.1. - [Amazon EMR 7.14 is available with Spark 3.5.8 and Iceberg 1.10.1](https://sparkaccelerationcompare.com/news/emr-7-14-available/index.md): AWS announced Amazon EMR 7.14 on September 22, 2026. - [Amazon EMR on EKS cuts the EMR charge by 30 percent](https://sparkaccelerationcompare.com/news/emr-eks-charge-cut-30-percent/index.md): AWS announced on October 23, 2026 a 30 percent reduction in the EMR charge added to EKS when running EMR on EKS. - [EMR Serverless raises the Spark shuffle limit from 200 GB to 1 TB per job](https://sparkaccelerationcompare.com/news/emr-serverless-terabyte-shuffle/index.md): AWS announced on October 1, 2026 that Amazon EMR Serverless supports up to 1 TB of shuffle per job, raising the previous 200 GB limit. - [Amazon EMR on EKS adds interactive Spark sessions with Spark Connect](https://sparkaccelerationcompare.com/news/emr-eks-spark-connect-interactive/index.md): AWS announced on September 24, 2026 that Amazon EMR on EKS supports interactive Apache Spark sessions through Spark Connect. - [Google Lightning Engine runs on Arm with Axion C4A for Dataproc 2.3 clusters](https://sparkaccelerationcompare.com/news/lightning-engine-arm-axion/index.md): Google's Dataproc release notes of September 1, 2026 say Lightning Engine with Native Query Execution is supported on Arm for image version 2.3 clusters (2.3-ubuntu2 - [Google Lightning Engine reaches Dataproc 3.0 cluster images from 3.0.2](https://sparkaccelerationcompare.com/news/lightning-engine-3-0-images/index.md): Google's release notes dated September 23, 2026 say Lightning Engine is supported in Dataproc 3.0 cluster images from subminor version 3.0.2 (3.0.2-debian13 and 3.0. - [DualBird blog Part 2: why data skew costs less on its FPGA architecture](https://sparkaccelerationcompare.com/news/dualbird-skew-part-2/index.md): DualBird published Part 2 of its Faster Spark series on September 1, 2026, written by its chief architect and co-founder. - [Flarion blog: the hard part of a Spark upgrade is proving the output did not change](https://sparkaccelerationcompare.com/news/flarion-spark-upgrade-validation/index.md): Flarion published a blog post on August 18, 2026 arguing that validation, not code rewriting, is the main obstacle to Spark version upgrades. ## Faq - [Frequently asked questions](https://sparkaccelerationcompare.com/faq/index.md): 30 questions with sourced answers. ## Situation - [Spark bill doubled in a year on EMR or EKS](https://sparkaccelerationcompare.com/situations/bill-doubled/index.md): Five checks for a data platform team on EMR or EKS whose Spark bill doubled, each answered with published, dated facts from the twelve options. - [Nightly pipelines that miss the morning](https://sparkaccelerationcompare.com/situations/late-pipelines/index.md): Five checks for a team whose nightly Spark pipelines overrun, with the runtimes and fallback limits each vendor has published. - [Data preparation for AI and feature pipelines](https://sparkaccelerationcompare.com/situations/ai-data-prep/index.md): Five checks for a team whose AI and feature pipelines wait on Spark data preparation, covering workload claims, UDFs, file formats, hardware and cloud. - [Leaving a managed platform for an open stack](https://sparkaccelerationcompare.com/situations/leaving-managed/index.md): Five checks for a team moving Spark from a managed platform to an open stack: what is lost, what runs outside, what you operate, license terms and versions. - [Paying every day to compact Iceberg tables](https://sparkaccelerationcompare.com/situations/iceberg-compaction/index.md): Five checks for a team that pays for Iceberg compaction every day: published compaction results, Iceberg support, cost per terabyte, test design and location. ## Datasets - [Datasets](https://sparkaccelerationcompare.com/datasets/index.md): The site's datasets as CSV and JSON, edition 10 Nov 2026.