# Spark Acceleration Compare: every page as markdown Twelve ways to speed up Spark without rewriting jobs, with a Ledger of published benchmarks and a Change Budget of what each makes a team change. --- # Ways to speed up Spark without rewriting your jobs, compared Side by side, read 10 Nov 2026 **The short answer.** Spark Acceleration Compare lists twelve options for making Apache Spark faster without rewriting jobs: plugins, native engines, GPU, FPGA and custom-silicon accelerators, and platform runtimes. It holds a Ledger of 21 published benchmarks and a Change Budget of what each option makes a team change. It lists the options and does not rank them. - 2026-11-03: Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 (run by Apache DataFusion Comet project) - 2026-11-03: Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 (run by Apache DataFusion Comet project) - 2026-09: Flarion TPC-DS SF100, six engines on one machine (run by Flarion) - 2026-09: Flarion TPC-DS SF1000, six engines on one machine (run by Flarion) ## The Ledger: the 10 newest of 21 published benchmarks | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | | 2026-08-25 | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) | | 2026-06-11 | [DualBird](/options/dualbird/) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found | | 2026-06-11 | [Speedata](/options/speedata/) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched | | 2025-08-21 | [DataPelago](/options/datapelago/) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No | | 2025-05-30 | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No | | 2022 | [Databricks Photon](/options/databricks-photon/) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No | Ordered by publication date, newest first. 20 of 21 entries were run by the maker of the option tested (its vendor or its own project). Undated entries come last. Full list: /ledger/. ## Four ways into the facts ### Asset: Spark Accelerator Ledger (21 entries, 1 run by another party) Every published benchmark for each option, with who ran it, on what, and the result as published. [Open the Ledger](/ledger/) ### Asset: Change Budget (12 options, 4 columns) What each option makes a team change: job code, the cluster, the cloud, and the vendor contract. [Open the Change Budget](/change-budget/) ### Side by side: Any two options (66 pairs) Two options in the same rows, with sources, and no conclusion drawn for you. [Pick a pair](/compare/) ### Profiles: The options, A to Z (12 options) Same fields in the same order for every option: what it does well, where it is limited, the published facts. [Read the profiles](/options/) ## What each option does well, and where it is limited | Option | Kind | Does well | Limited | | --- | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS-tuned Spark runtime on EMR | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. | | [Apache Auron](/options/apache-auron/) | Open-source Rust engine plugin for Spark | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. | | [Apache DataFusion Comet](/options/datafusion-comet/) | Open-source Spark plugin on DataFusion | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Open-source Spark plugin offloading to Velox | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. | | [Databricks Photon](/options/databricks-photon/) | Databricks's C++ vectorized engine | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. | | [DataPelago](/options/datapelago/) | Gluten-based CPU and GPU accelerator | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. | | [DualBird](/options/dualbird/) | FPGA Spark plugin on AWS F2 | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. | | [Flarion](/options/flarion/) | Rust and DataFusion engine as one JAR | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Premium Spark engine on Google Cloud | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Velox-based engine inside Fabric Spark | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | GPU plugin, formerly RAPIDS Accelerator | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. | | [Speedata](/options/speedata/) | PCIe analytics chip plus Dash Spark plugin | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. | Alphabetical. First sentence of each; the full text and sources are on the option's profile. ## Five situations - [Spark bill doubled in a year on EMR or EKS](/situations/bill-doubled/): Five checks for a data platform team on EMR or EKS whose Spark bill doubled, each answered with published, dated facts from the twelve options. - [Nightly pipelines that miss the morning](/situations/late-pipelines/): Five checks for a team whose nightly Spark pipelines overrun, with the runtimes and fallback limits each vendor has published. - [Data preparation for AI and feature pipelines](/situations/ai-data-prep/): Five checks for a team whose AI and feature pipelines wait on Spark data preparation, covering workload claims, UDFs, file formats, hardware and cloud. - [Leaving a managed platform for an open stack](/situations/leaving-managed/): Five checks for a team moving Spark from a managed platform to an open stack: what is lost, what runs outside, what you operate, license terms and versions. - [Paying every day to compact Iceberg tables](/situations/iceberg-compaction/): Five checks for a team that pays for Iceberg compaction every day: published compaction results, Iceberg support, cost per terabyte, test design and location. ## What Spark acceleration means here Spark acceleration, on this site, means a product or project that makes existing Apache Spark jobs run faster or cheaper while leaving the job code in place. The usual form is a plugin or a runtime setting that takes over the operators it supports, such as scans, joins and aggregations, and hands the rest back to Spark. The twelve options fall into four groups. Native-engine plugins (Apache Auron, Apache DataFusion Comet, Apache Gluten with Velox, Flarion, DataPelago) run supported operators in C++ or Rust. Hardware accelerators (the NVIDIA cuDF plugin on GPUs, DualBird on FPGAs, Speedata on its own chip) move work off the CPU. Platform runtimes (Databricks Photon, the Amazon EMR runtime, Google Cloud Lightning Engine, the Microsoft Fabric native execution engine) are tied to one provider's service. Options that need the jobs rewritten, such as moving from Spark to another engine or from RDD code to SQL, are outside this site. Each option page records what the vendor says it changes, where it runs and what has been published about its speed and price, with a source and a read date on every fact. Data catalogs, lineage, semantic layers and MCP servers for data are outside this site's subject. ## Guides and glossary - [Where a Spark job spends its time: scan to JVM](/guides/where-spark-time-goes/) - [What "no code change" means for Spark acceleration](/guides/what-no-code-change-means/) - [How to read a vendor Spark benchmark: seven checks](/guides/reading-a-vendor-benchmark/) - [GPU, FPGA and custom silicon for Spark in plain words](/guides/hardware-acceleration-plain-words/) - [Spark performance tuning: what to do when it is not enough](/guides/when-tuning-is-cheaper/) - [Native Spark engines: Auron, DataFusion, Photon, Velox](/guides/native-engines-explained/) - [All guides](/guides/) - [Glossary: 35 terms](/glossary/) ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ --- # 12 ways to speed up Spark, listed A to Z with the same fields Options, read 10 Nov 2026 **The short answer.** The twelve options covered, in alphabetical order: Amazon EMR runtime, Apache Auron, Apache DataFusion Comet, Apache Gluten with Velox, Databricks Photon, DataPelago, DualBird, Flarion, Google Cloud Lightning Engine, Microsoft Fabric native execution engine, the NVIDIA cuDF plugin and Speedata. Each page gives sourced facts, dates and what is not found. ## Every option on this site | Option | Kind | Ledger entries | Summary | | --- | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS-tuned Spark runtime on EMR | 1 | Spark runtime from AWS: 100% API compatible per AWS, on EMR on EC2, EKS and Serverless, with a public TPC-DS harness. | | [Apache Auron](/options/apache-auron/) | Open-source Rust engine plugin for Spark | 1 | Apache-2.0 Spark plugin with a Rust native engine, a TPC-DS 1 TB method and builds for Spark 3.0 to 4.2. | | [Apache DataFusion Comet](/options/datafusion-comet/) | Open-source Spark plugin on DataFusion | 2 | Apache-2.0 Spark plugin on DataFusion with published TPC-DS scripts, settings and monthly releases since 1.0.0. | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Open-source Spark plugin offloading to Velox | 2 | Apache-2.0 Spark plugin offloading to Velox, with a built-in benchmark tool and a self-reported adopter list. | | [Databricks Photon](/options/databricks-photon/) | Databricks's C++ vectorized engine | 2 | Databricks's own C++ engine, on by default in classic jobs compute and documented only inside Databricks. | | [DataPelago](/options/datapelago/) | Gluten-based CPU and GPU accelerator | 1 | Proprietary Gluten-based Spark plugin for EKS, Dataproc and GKE, billed through cloud marketplaces. | | [DualBird](/options/dualbird/) | FPGA Spark plugin on AWS F2 | 3 | FPGA-based Spark plugin for AWS F2 instances, with one published worked example, an Iceberg compaction test. | | [Flarion](/options/flarion/) | Rust and DataFusion engine as one JAR | 2 | Proprietary native Spark engine as one JAR, with published per-core list prices and a six-engine TPC-DS page. | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Premium Spark engine on Google Cloud | 3 | Google Cloud's Spark engine tier with optional native execution, priced per DCU-hour in us-central1. | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Velox-based engine inside Fabric Spark | 0 | Velox and Gluten engine inside Fabric Spark on Azure, switched on with one toggle or one setting. | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | GPU plugin, formerly RAPIDS Accelerator | 2 | GPU plugin for Spark SQL and DataFrame jobs, with a cloud-provider benchmark that publishes runtime and cost per run. | | [Speedata](/options/speedata/) | PCIe analytics chip plus Dash Spark plugin | 2 | Custom-silicon Spark offload on a PCIe card, sold as servers, OEM hardware or a hosted AWS service. | Alphabetical by name. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/ --- # Amazon EMR runtime for Apache Spark, AWS-tuned Spark runtime on EMR: the published facts Option profile, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark is AWS's performance-optimized build of Spark, offered on EMR on EC2, EMR on EKS and EMR Serverless. AWS says it is 100% API compatible with open source Spark and runs up to 4.5x faster. AWS also publishes a TPC-DS benchmark harness on GitHub that uses a 3 TB dataset. - **Kind:** AWS-tuned Spark runtime on EMR - **Website:** [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/) - **Name checked:** Still offered as the performance-optimized Amazon EMR runtime for Apache Spark. AWS pages describe it as 100% API compatible with open source Spark and available on EMR on EC2, EMR on EKS and EMR Serverless. - **Read:** 10 Nov 2026 ## What it does well - AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. - AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. ## Where it is limited - The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. - AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | vendor docs | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | vendor docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | vendor docs | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | cloud provider docs | [docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | vendor docs | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | vendor docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. | vendor | [github.com/aws-samples/emr-spark-benchmark](https://github.com/aws-samples/emr-spark-benchmark), 10 Nov 2026 | | The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. | vendor | [github.com/aws-samples/emr-spark-benchmark](https://github.com/aws-samples/emr-spark-benchmark), 10 Nov 2026 | | A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | vendor | [github.com/aws-samples/emr-on-eks-benchmark](https://github.com/aws-samples/emr-on-eks-benchmark), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | The runtime is proprietary to AWS and is a tuned build of Apache Spark, not a native C++ replacement. The benchmark harness is open sample code on GitHub. | vendor | [github.com/aws-samples/emr-spark-benchmark](https://github.com/aws-samples/emr-spark-benchmark), 10 Nov 2026 | ## Ledger entries for Amazon EMR runtime for Apache Spark | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | Not stated | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) | AWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performance | AWS (its maker) | Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code change stated | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | Cluster | Run the jobs on an EMR release | Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | [Amazon EMR Spark performance guide](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 | | Cloud | AWS only | AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | Vendor | EMR charges added to EC2 and EBS | The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - AWS Big Data blog post with TPC-DS 3 TB per-version results (EMR runtime vs OSS Spark) and cost-for-same-work figures (searched: aws.amazon.com/blogs/big-data/amazon-emr-on-eks-widens-the-performance-g, aws.amazon.com/blogs/big-data/category/analytics/amazon-emr/ (category p, github.com/aws-samples/emr-spark-benchmark) - Published cost-for-the-same-work result for EMR runtime vs OSS Spark (searched: aws.amazon.com/emr/pricing, aws.amazon.com/emr/features/spark, github.com/aws-samples/emr-spark-benchmark) ## Amazon EMR runtime for Apache Spark side by side with each other option - [Amazon EMR runtime for Apache Spark and Apache Auron](/compare/apache-auron-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Apache DataFusion Comet](/compare/datafusion-comet-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Databricks Photon](/compare/databricks-photon-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and DataPelago](/compare/datapelago-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and DualBird](/compare/dualbird-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Flarion](/compare/emr-spark-runtime-vs-flarion/) - [Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine](/compare/emr-spark-runtime-vs-fabric-native-execution-engine/) - [Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark](/compare/emr-spark-runtime-vs-nvidia-rapids-spark/) - [Amazon EMR runtime for Apache Spark and Speedata](/compare/emr-spark-runtime-vs-speedata/) ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/emr-spark-runtime/ --- # Apache Auron, Open-source Rust engine plugin for Spark: the published facts Option profile, read 10 Nov 2026 **The short answer.** Apache Auron, formerly Blaze, is an Apache-2.0 Spark plugin with a native engine written in Rust on Apache DataFusion and Arrow. Its homepage claims about 2x faster than Spark 3.5 with about 50% fewer cluster resources. Its build script lists Spark 3.0 to 4.2, and release 8.0.0-incubating is dated 2026-07-18. - **Kind:** Open-source Rust engine plugin for Spark - **Website:** [auron.apache.org](https://auron.apache.org/) - **Name checked:** Renamed from Blaze to Auron. Releases up to 4.0.1 (Dec 2024) shipped as 'blaze-engine' jars; 5.0.0 (Apr 28 2025) and later are Auron. The GitHub repo kwai/blaze now resolves to apache/auron. Releases 6.0.0, 7.0.0 and 8.0.0 are labeled 'incubating', but the project homepage footer calls it a top-level project, so the project's own pages disagree on status. - **Read:** 10 Nov 2026 ## What it does well - The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. - Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. ## Where it is limited - You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. - Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. - The benchmark's per-query results are published as a chart image only. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | open source project | [auron.apache.org](https://auron.apache.org/), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | open source project | [auron.apache.org](https://auron.apache.org/), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | open source project | [auron.apache.org/documents/benchmarks.html](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. | open source project | [raw.githubusercontent.com/apache/auron/master/auron-build.sh](https://raw.githubusercontent.com/apache/auron/master/auron-build.sh), 10 Nov 2026 | | The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | open source project | [auron.apache.org/archives/v8.0.0-incubating.html](https://auron.apache.org/archives/v8.0.0-incubating.html), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | open source project | [raw.githubusercontent.com/apache/auron/master/README.md](https://raw.githubusercontent.com/apache/auron/master/README.md), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | open source project | [auron.apache.org/documents/benchmarks.html](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | Apache-2.0 licensed under the Apache Software Foundation; GitHub repo (originally kwai/blaze) created 2021-06-28, about 1,800 stars. The homepage says it is verified in production with exabytes of data; that claim names no company. | open source project | [auron.apache.org](https://auron.apache.org/), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Releases: 8.0.0-incubating (2026-07-18), 7.0.0-incubating (2026-03-04), 6.0.0-incubating (2025-10-17), 5.0.0 (2025-04-28), 4.0.1 (2024-12-10), 4.0.0 (2024-10-10), going back to 2.0.6 in Sep 2023 under the Blaze name. | open source project | [auron.apache.org/archives/all-releases.html](https://auron.apache.org/archives/all-releases.html), 10 Nov 2026 | ## Ledger entries for Apache Auron | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | Not stated | [Apache Auron](/options/apache-auron/) | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) | Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image only | Apache Auron project (its maker) | Yes, tools and full Spark config in the auron-project/tpcds-tools repository | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No job code change stated | Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | [Apache Auron homepage](https://auron.apache.org/), 10 Nov 2026 | | Cluster | Build the jar, set three configs | You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | Cloud | None found on 2026-11-10 | No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | [Apache Auron README](https://raw.githubusercontent.com/apache/auron/master/README.md), 10 Nov 2026 | | Vendor | Apache-2.0, no contract stated | Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | [Apache Auron homepage](https://auron.apache.org/), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Any published cost per job or dollar cost-for-the-same-work result (only a relative 'about 50% cluster resources' claim) (searched: auron.apache.org, auron.apache.org/documents/benchmarks.html, raw.githubusercontent.com/apache/auron/master/README.md) - Confirmation that Kuaishou created Blaze or still contributes (not stated on the fetched pages; only the kwai GitHub organization name hints at it) (searched: auron.apache.org, raw.githubusercontent.com/apache/auron/master/README.md, auron.apache.org/archives/v4.0.0.html) - Documented support or tested configuration for Amazon EMR, EKS or Databricks (searched: auron.apache.org/documents/getting-started.html, raw.githubusercontent.com/apache/auron/master/README.md) ## Apache Auron side by side with each other option - [Apache Auron and Amazon EMR runtime for Apache Spark](/compare/apache-auron-vs-emr-spark-runtime/) - [Apache Auron and Apache DataFusion Comet](/compare/apache-auron-vs-datafusion-comet/) - [Apache Auron and Apache Gluten with Velox](/compare/apache-auron-vs-apache-gluten-velox/) - [Apache Auron and Databricks Photon](/compare/apache-auron-vs-databricks-photon/) - [Apache Auron and DataPelago](/compare/apache-auron-vs-datapelago/) - [Apache Auron and DualBird](/compare/apache-auron-vs-dualbird/) - [Apache Auron and Flarion](/compare/apache-auron-vs-flarion/) - [Apache Auron and Google Cloud Lightning Engine](/compare/apache-auron-vs-dataproc-lightning-engine/) - [Apache Auron and Microsoft Fabric native execution engine](/compare/apache-auron-vs-fabric-native-execution-engine/) - [Apache Auron and NVIDIA cuDF plugin for Apache Spark](/compare/apache-auron-vs-nvidia-rapids-spark/) - [Apache Auron and Speedata](/compare/apache-auron-vs-speedata/) ## Sources - Apache Auron: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Apache Auron: auron.apache.org/archives/all-releases.html: https://auron.apache.org/archives/all-releases.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/apache-auron/ --- # Apache DataFusion Comet, Open-source Spark plugin on DataFusion: the published facts Option profile, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet is an Apache-2.0 Spark plugin that runs supported operators on the Apache DataFusion native engine, written in Rust. The project's TPC-DS run at scale factor 1000 on an EKS cluster took 994 seconds on Spark 4.2.0 and 488 seconds on Comet 1.2.0. Release 1.2.0 shipped on 2026-11-03. - **Kind:** Open-source Spark plugin on DataFusion - **Website:** [datafusion.apache.org/comet](https://datafusion.apache.org/comet/) - **Name checked:** Exists under the same name, Apache DataFusion Comet, as a subproject of Apache DataFusion (apache/datafusion-comet). No rename found. - **Read:** 10 Nov 2026 ## What it does well - The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. - Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. ## Where it is limited - The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. - Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | open source project | [datafusion.apache.org/comet/contributor-guide/benchmarking.html](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | Licensed Apache-2.0 under the Apache Software Foundation; the docs site footer shows copyright 2023 to 2026 Apache Software Foundation. GitHub repo created 2024-01-15, about 1,300 stars. | open source project | [api.github.com/repos/apache/datafusion-comet](https://api.github.com/repos/apache/datafusion-comet), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | GitHub releases: 1.0.0 (2026-08-07), 1.1.0 (2026-10-06), 1.2.0 (2026-11-03), roughly monthly after 1.0. | open source project | [api.github.com/repos/apache/datafusion-comet/releases?per_page=3](https://api.github.com/repos/apache/datafusion-comet/releases?per_page=3), 10 Nov 2026 | ## Ledger entries for Apache DataFusion Comet | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No job code change stated | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | [Comet compatibility guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 | | Cluster | Plugin, shuffle manager, off-heap memory | Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | [Comet tuning guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md), 10 Nov 2026 | | Cloud | None found on 2026-11-10 | No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | Vendor | Apache-2.0, no contract stated | Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | [Apache DataFusion Comet GitHub repository](https://api.github.com/repos/apache/datafusion-comet), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Any published cost per job or cost-for-the-same-work result (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, datafusion.apache.org/comet/contributor-guide/benchmarking_aws_ec2.html) - Company sponsors or major corporate contributors (the fetched pages name only the Apache Software Foundation) (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, api.github.com/repos/apache/datafusion-comet) ## Apache DataFusion Comet side by side with each other option - [Apache DataFusion Comet and Amazon EMR runtime for Apache Spark](/compare/datafusion-comet-vs-emr-spark-runtime/) - [Apache DataFusion Comet and Apache Auron](/compare/apache-auron-vs-datafusion-comet/) - [Apache DataFusion Comet and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-datafusion-comet/) - [Apache DataFusion Comet and Databricks Photon](/compare/databricks-photon-vs-datafusion-comet/) - [Apache DataFusion Comet and DataPelago](/compare/datafusion-comet-vs-datapelago/) - [Apache DataFusion Comet and DualBird](/compare/datafusion-comet-vs-dualbird/) - [Apache DataFusion Comet and Flarion](/compare/datafusion-comet-vs-flarion/) - [Apache DataFusion Comet and Google Cloud Lightning Engine](/compare/datafusion-comet-vs-dataproc-lightning-engine/) - [Apache DataFusion Comet and Microsoft Fabric native execution engine](/compare/datafusion-comet-vs-fabric-native-execution-engine/) - [Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark](/compare/datafusion-comet-vs-nvidia-rapids-spark/) - [Apache DataFusion Comet and Speedata](/compare/datafusion-comet-vs-speedata/) ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) - Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet/releases?per_page=3: https://api.github.com/repos/apache/datafusion-comet/releases?per_page=3 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/datafusion-comet/ --- # Apache Gluten with Velox, Open-source Spark plugin offloading to Velox: the published facts Option profile, read 10 Nov 2026 **The short answer.** Apache Gluten is an Apache-2.0 Spark plugin that offloads execution to Meta's Velox C++ library. Its homepage reports 3.34x overall on TPC-H and 3.02x on TPC-DS at 3 TB, on Intel Xeon Platinum 8592+ with Spark 3.3.1. The Powered By page lists adopters including Alibaba, Microsoft (Fabric) and Google (Lightning engine). - **Kind:** Open-source Spark plugin offloading to Velox - **Website:** [gluten.apache.org](https://gluten.apache.org/) - **Name checked:** The project is now called Apache Gluten and its homepage describes it as an Apache Top-Level Project. The GitHub repo apache/incubator-gluten now resolves to apache/gluten. Velox is Meta's C++ library; a Gluten release note says the Velox fork reference moved from oap-project to IBM. - **Read:** 10 Nov 2026 ## What it does well - Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. - The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. ## Where it is limited - The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. - Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | open source project | [gluten.apache.org](https://gluten.apache.org/), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. | open source project | [raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st](https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md), 10 Nov 2026 | | Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | open source project | [raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | open source project | [raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st](https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. | open source project | [raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md), 10 Nov 2026 | | Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | open source project | [raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | open source project | [gluten.apache.org/downloads](https://gluten.apache.org/downloads/), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | open source project | [raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md](https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | The Powered By page lists adopters including Alibaba, Baidu, Bytedance, BIGO, IBM, Kyligence, Meituan, Microsoft (Fabric), Google (Lightning engine), Netease, Oracle (OCI), Palantir, Pinterest, SF Express and Tencent. These are self-reported entries. The homepage benchmark runs on Intel hardware; the docs describe Velox as Meta's C++ execution library. Project license is Apache-2.0 (GitHub API). | open source project | [gluten.apache.org/poweredby](https://gluten.apache.org/poweredby/), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | GitHub releases: v1.7.0 published 2026-08-26 (rc1 2026-08-17). The downloads page lists 1.7.0 as 'Jun. 6 2025', which conflicts with the GitHub date and looks like a typo; 1.6.0 is listed as Mar. 10 2026 and 1.5.0 as Oct. 29 2025. Repo created 2021-12-06, about 1,600 stars. | open source project | [api.github.com/repos/apache/incubator-gluten/releases?per_page=3](https://api.github.com/repos/apache/incubator-gluten/releases?per_page=3), 10 Nov 2026 | ## Ledger entries for Apache Gluten with Velox | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) | Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queries | Apache Gluten project (its maker) | No | | Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) | Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queries | Apache Gluten project (its maker) | No | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No DataFrame or SQL changes | Gluten needs no DataFrame or SQL changes, only configuration. | [Apache Gluten getting started](https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md), 10 Nov 2026 | | Cluster | Plugin, off-heap memory, shuffle manager | The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | [Apache Gluten getting started](https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md), 10 Nov 2026 | | Cloud | None found on 2026-11-10 | No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | [Apache Gluten downloads](https://gluten.apache.org/downloads/), 10 Nov 2026 | | Vendor | Apache-2.0, no contract stated | Gluten's project license is Apache-2.0; no vendor or platform contract is described. | [Apache Gluten Powered By page](https://gluten.apache.org/poweredby/), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Any published cost per job or cost-for-the-same-work result (the software is free under Apache-2.0) (searched: gluten.apache.org, raw.githubusercontent.com/apache/gluten/main/docs/index.md, raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st) - Documented support or tested configuration for Amazon EMR, EKS or Databricks (searched: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, gluten.apache.org/poweredby, raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st) - An independent benchmark of Gluten on AWS hardware (searched: gluten.apache.org, gluten.apache.org/poweredby, gluten.apache.org/community) ## Apache Gluten with Velox side by side with each other option - [Apache Gluten with Velox and Amazon EMR runtime for Apache Spark](/compare/apache-gluten-velox-vs-emr-spark-runtime/) - [Apache Gluten with Velox and Apache Auron](/compare/apache-auron-vs-apache-gluten-velox/) - [Apache Gluten with Velox and Apache DataFusion Comet](/compare/apache-gluten-velox-vs-datafusion-comet/) - [Apache Gluten with Velox and Databricks Photon](/compare/apache-gluten-velox-vs-databricks-photon/) - [Apache Gluten with Velox and DataPelago](/compare/apache-gluten-velox-vs-datapelago/) - [Apache Gluten with Velox and DualBird](/compare/apache-gluten-velox-vs-dualbird/) - [Apache Gluten with Velox and Flarion](/compare/apache-gluten-velox-vs-flarion/) - [Apache Gluten with Velox and Google Cloud Lightning Engine](/compare/apache-gluten-velox-vs-dataproc-lightning-engine/) - [Apache Gluten with Velox and Microsoft Fabric native execution engine](/compare/apache-gluten-velox-vs-fabric-native-execution-engine/) - [Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark](/compare/apache-gluten-velox-vs-nvidia-rapids-spark/) - [Apache Gluten with Velox and Speedata](/compare/apache-gluten-velox-vs-speedata/) ## Sources - Apache Gluten with Velox: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with Velox: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with Velox: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with Velox: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with Velox: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with Velox: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with Velox: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Apache Gluten with Velox: gluten.apache.org/poweredby: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) - Apache Gluten with Velox: api.github.com/repos/apache/incubator-gluten/releases?per_page=3: https://api.github.com/repos/apache/incubator-gluten/releases?per_page=3 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/apache-gluten-velox/ --- # Databricks Photon, Databricks's C++ vectorized engine: the published facts Option profile, read 10 Nov 2026 **The short answer.** Photon is the C++ vectorized query engine inside Databricks compute, documented on AWS, Azure and Google Cloud. Databricks's documentation says it works with existing Spark API code and is on by default for classic jobs compute. Databricks announced an audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. - **Kind:** Databricks's C++ vectorized engine - **Website:** [www.databricks.com](https://www.databricks.com/) - **Name checked:** Still called Photon in current Databricks docs (page last updated Sep 11, 2026). Documented as the Databricks-native vectorized query engine; the page describes it only inside Databricks compute (serverless, SQL warehouses, classic all-purpose and jobs compute, Lakeflow pipelines). No offering outside Databricks was found on this page. - **Read:** 10 Nov 2026 ## What it does well - Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. - Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. ## Where it is limited - Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. - Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. | vendor | [people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf), 10 Nov 2026 | | Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | vendor | [databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | vendor docs | [databricks.com/product/pricing](https://www.databricks.com/product/pricing), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | vendor | [databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | Photon is proprietary to Databricks. It was described in a SIGMOD 2022 paper by Databricks engineers. Source code is not open. | vendor | [people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Photon is the Databricks-native vectorized query engine. It replaces the JVM Spark SQL execution layer with a C++ runtime for supported operations while Catalyst still plans the query. | vendor docs | [docs.databricks.com/en/compute/photon.html](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | ## Ledger entries for Databricks Photon | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2022 | [Databricks Photon](/options/databricks-photon/) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No | | 2021-11-02 | [Databricks Photon](/options/databricks-photon/) | [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) | 32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba) | Databricks, audited by the TPC (its maker) | No (a TPC audit report exists under TPC rules; not a repeatable open script) | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No change, with listed exceptions | Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | Cluster | Set runtime_engine to PHOTON in API | Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | Cloud | Databricks on AWS, Azure or GCP | Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | Vendor | Databricks platform required | Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | [Databricks pricing page](https://www.databricks.com/product/pricing), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Independent third-party benchmark of Photon vs open source Spark on batch ETL (searched: docs.databricks.com/en/compute/photon.html, databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing, people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf) ## Databricks Photon side by side with each other option - [Databricks Photon and Amazon EMR runtime for Apache Spark](/compare/databricks-photon-vs-emr-spark-runtime/) - [Databricks Photon and Apache Auron](/compare/apache-auron-vs-databricks-photon/) - [Databricks Photon and Apache DataFusion Comet](/compare/databricks-photon-vs-datafusion-comet/) - [Databricks Photon and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-databricks-photon/) - [Databricks Photon and DataPelago](/compare/databricks-photon-vs-datapelago/) - [Databricks Photon and DualBird](/compare/databricks-photon-vs-dualbird/) - [Databricks Photon and Flarion](/compare/databricks-photon-vs-flarion/) - [Databricks Photon and Google Cloud Lightning Engine](/compare/databricks-photon-vs-dataproc-lightning-engine/) - [Databricks Photon and Microsoft Fabric native execution engine](/compare/databricks-photon-vs-fabric-native-execution-engine/) - [Databricks Photon and NVIDIA cuDF plugin for Apache Spark](/compare/databricks-photon-vs-nvidia-rapids-spark/) - [Databricks Photon and Speedata](/compare/databricks-photon-vs-speedata/) ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/databricks-photon/ --- # DataPelago, Gluten-based CPU and GPU accelerator: the published facts Option profile, read 10 Nov 2026 **The short answer.** DataPelago Accelerator for Spark (DPA-S), launched in August 2025, is a proprietary Gluten-based plugin on the company's Nucleus engine. Per its documentation it runs on EKS in AWS and on Dataproc and GKE in Google Cloud. DataPelago claims 10X faster at 80% lower cost, with no workload or method attached. - **Kind:** Gluten-based CPU and GPU accelerator - **Website:** [www.datapelago.ai](https://www.datapelago.ai/) - **Name checked:** DataPelago, Inc. (Mountain View, CA) is live. datapelago.com redirects to www.datapelago.ai; page links also reference datapelago.io. The Spark product is named DataPelago Accelerator for Spark (DPA-S), launched Aug 2025. - **Read:** 10 Nov 2026 ## What it does well - Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. - DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. ## Where it is limited - DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. - Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. - The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. | vendor | [datapelago.ai](https://www.datapelago.ai/), 10 Nov 2026 | | DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | vendor | [datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF](https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. | vendor | [datapelago.ai](https://www.datapelago.ai/), 10 Nov 2026 | | The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | vendor | [datapelago.ai/dpa](https://www.datapelago.ai/dpa), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | vendor | [datapelago.ai/dpa](https://www.datapelago.ai/dpa), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | vendor docs | [datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. | vendor docs | [datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | | GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. | vendor docs | [datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html), 10 Nov 2026 | | The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | vendor | [datapelago.ai](https://www.datapelago.ai/), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | vendor | [datapelago.ai/technology](https://www.datapelago.ai/technology), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Founder and CEO Rajan Goyal was previously CTO at Fungible. DataPelago, Inc. is at 100 View Street, Mountain View, CA. Timeline from the press page: Universal Data Processing Engine unveiled Oct 1, 2024; President John Chirapurath appointed Jul 28, 2025; DPA-S launched Aug 5, 2025; Nucleus-versus-cuDF release Aug 22, 2025; Fast Company Most Innovative Companies of 2026 (Mar 24, 2026); Data Breakthrough award (May 13, 2026). Investor/partner logos on the home page include Taiwania, Qualcomm, Eclipse, Alter and Nautilus (alt text); no round sizes were found. | vendor | [datapelago.ai/press](https://www.datapelago.ai/press), 10 Nov 2026 | ## Ledger entries for DataPelago | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2025-08-21 | [DataPelago](/options/datapelago/) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No application rewrite stated | DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | Cluster | Plugin, license and memory settings | Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | [DataPelago AWS deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | | Cloud | AWS EKS or Google Cloud | DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | [DataPelago Google Cloud deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html), 10 Nov 2026 | | Vendor | Marketplace subscription required | DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | [DataPelago AWS deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - TPC-DS/TPC-H or end-to-end Spark benchmark with published method, and any third-party benchmark (searched: datapelago.ai, datapelago.ai/dpa, datapelago.ai/technology) - Funding amounts, round dates and lead investors (searched: datapelago.ai/company, datapelago.ai/press, datapelago.ai) - Published unit prices (searched: datapelago.ai/dpa, datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html) ## DataPelago side by side with each other option - [DataPelago and Amazon EMR runtime for Apache Spark](/compare/datapelago-vs-emr-spark-runtime/) - [DataPelago and Apache Auron](/compare/apache-auron-vs-datapelago/) - [DataPelago and Apache DataFusion Comet](/compare/datafusion-comet-vs-datapelago/) - [DataPelago and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-datapelago/) - [DataPelago and Databricks Photon](/compare/databricks-photon-vs-datapelago/) - [DataPelago and DualBird](/compare/datapelago-vs-dualbird/) - [DataPelago and Flarion](/compare/datapelago-vs-flarion/) - [DataPelago and Google Cloud Lightning Engine](/compare/datapelago-vs-dataproc-lightning-engine/) - [DataPelago and Microsoft Fabric native execution engine](/compare/datapelago-vs-fabric-native-execution-engine/) - [DataPelago and NVIDIA cuDF plugin for Apache Spark](/compare/datapelago-vs-nvidia-rapids-spark/) - [DataPelago and Speedata](/compare/datapelago-vs-speedata/) ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - DataPelago: datapelago.ai/press: https://www.datapelago.ai/press (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/datapelago/ --- # DualBird, FPGA Spark plugin on AWS F2: the published facts Option profile, read 10 Nov 2026 **The short answer.** DualBird sells a Spark plugin that runs on AWS EC2 F2 instances, which carry AMD Virtex UltraScale+ FPGAs. DualBird says it delivers 10x to 30x faster processing and 50% to 90% lower costs with no changes required. Onboarding, per DualBird, is two steps: switch the instance type to F2 and add the plugin. - **Kind:** FPGA Spark plugin on AWS F2 - **Website:** [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile) - **Name checked:** Site dualbird.io is live under the name DualBird (written 'Dualbird' in the footer). dualbird.net serves the same site. No rename or acquisition found. - **Read:** 10 Nov 2026 ## What it does well - DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). - DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. ## Where it is limited - DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. - A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. | vendor | [dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | vendor | [dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | vendor | [dualbird.io/resources/blog/a-simpler-spark](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). | cloud provider docs | [aws.amazon.com/ec2/instance-types/f2](https://aws.amazon.com/ec2/instance-types/f2/), 10 Nov 2026 | | The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. | cloud provider docs | [docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | vendor | [dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper](https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | DualBird's own blog (Feb 5, 2026) announces a $25M raise led by Lightspeed Venture Partners with Angular Ventures, Uncork Capital and Bessemer Venture Partners. It says general availability was planned for early 2026 and that early deployments with design partners showed hours-to-minutes gains. The funding is confirmed by the company itself only. Lightspeed's page returned HTTP 403 and no press article was reachable. | vendor | [dualbird.io/resources/blog/dualbird-raises-25m](https://dualbird.io/resources/blog/dualbird-raises-25m?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Use cases on the home page: AI data infrastructure (feature engineering and AI data prep), big data pipelines, and accelerating data analytics (fresher data for dashboards). The Iceberg compaction benchmark is the one worked example with published method. | vendor | [dualbird.io](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | About page: founded and led by semiconductor and distributed data systems veterans; positions FPGA-based acceleration as a drop-in Spark plugin on standard AWS instances. Blog posts date from Dec 1, 2025 to Sep 1, 2026. Footer copyright reads 2025. Contact: contact@dualbird.io. | vendor | [dualbird.io/about](https://dualbird.io/about?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Ledger entries for DualBird | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-06-11 | [DualBird](/options/dualbird/) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found | | Not stated | [DualBird](/options/dualbird/) | [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) | DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not stated | DualBird (its maker) | No | | Not stated | [DualBird](/options/dualbird/) | [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) | DualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark) | DualBird (its maker) | No | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code change stated | DualBird's about page says 'No infrastructure changes. No code changes.' | [DualBird about page](https://dualbird.io/about?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Cluster | Instance type change to F2 | DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | [DualBird product page](https://dualbird.io/product?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Cloud | AWS only, F2 in eight regions | F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | Vendor | Plugin contract; no price published | No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Third-party (non-DualBird) benchmark or review of DualBird by another party (searched: dualbird.io/resources, github.com/dualbird, lsvp.com/dualbird) - Supported Spark versions, Databricks support, table format support list (searched: dualbird.io, dualbird.io/product, dualbird.io/about) - Published price or pricing model for the DualBird plugin (searched: dualbird.io, dualbird.io/product, aws.amazon.com/marketplace/search/results?searchTerms=dualbird) - Named customers (only unnamed design partners) and press coverage of the funding round (searched: dualbird.io, dualbird.io/resources/blog/dualbird-raises-25m, dualbird.io/team) ## DualBird side by side with each other option - [DualBird and Amazon EMR runtime for Apache Spark](/compare/dualbird-vs-emr-spark-runtime/) - [DualBird and Apache Auron](/compare/apache-auron-vs-dualbird/) - [DualBird and Apache DataFusion Comet](/compare/datafusion-comet-vs-dualbird/) - [DualBird and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-dualbird/) - [DualBird and Databricks Photon](/compare/databricks-photon-vs-dualbird/) - [DualBird and DataPelago](/compare/datapelago-vs-dualbird/) - [DualBird and Flarion](/compare/dualbird-vs-flarion/) - [DualBird and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-dualbird/) - [DualBird and Microsoft Fabric native execution engine](/compare/dualbird-vs-fabric-native-execution-engine/) - [DualBird and NVIDIA cuDF plugin for Apache Spark](/compare/dualbird-vs-nvidia-rapids-spark/) - [DualBird and Speedata](/compare/dualbird-vs-speedata/) ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/dualbird-raises-25m: https://dualbird.io/resources/blog/dualbird-raises-25m (read 10 Nov 2026) - DualBird: dualbird.io/about: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/dualbird/ --- # Flarion, Rust and DataFusion engine as one JAR: the published facts Option profile, read 10 Nov 2026 **The short answer.** Flarion is a proprietary native execution engine built on Apache Arrow and DataFusion, delivered as one JAR inside an existing Spark cluster. Its published list prices start at $0.009 per CPU hour for self-hosted Spark on AWS, Azure or Google Cloud. Its September 2026 benchmarks page compares six engines on TPC-DS. - **Kind:** Rust and DataFusion engine as one JAR - **Website:** [www.flarion.io](https://www.flarion.io/) - **Name checked:** flarion.io is live and redirects to www.flarion.io under the name Flarion. No rename found. - **Read:** 10 Nov 2026 ## What it does well - Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. - Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. ## Where it is limited - Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. - A published list of supported Spark versions: none found on 2026-11-10. - Third-party benchmarks of Flarion: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. | vendor | [flarion.io](https://www.flarion.io/), 10 Nov 2026 | | The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | vendor | [flarion.io/benchmarks](https://www.flarion.io/benchmarks), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | vendor | [flarion.io/pricing](https://www.flarion.io/pricing), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | vendor | [flarion.io/faqs](https://www.flarion.io/faqs), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | vendor | [flarion.io/faqs](https://www.flarion.io/faqs), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | vendor | [flarion.io/product](https://www.flarion.io/product), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | vendor | [flarion.io/faqs](https://www.flarion.io/faqs), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | vendor | [flarion.io/benchmarks](https://www.flarion.io/benchmarks), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Founders: Ran Reichman (CEO) and Udi Tizan (COO; the bio cites the Granulate $650M exit). Investor logos on the About page: Essence, Hetz Ventures, Sierra Ventures (read from image alt text; no amounts). The product is built on Apache Arrow and DataFusion, and the team says it contributes to those projects, but Flarion itself is proprietary. Footer copyright 2026. | vendor | [flarion.io/about-us](https://www.flarion.io/about-us), 10 Nov 2026 | ## Ledger entries for Flarion | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code change stated | Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | Cluster | One JAR and one config change | Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | Cloud | No single cloud required | Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | Vendor | Flarion license at usage-based prices | Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | [Flarion pricing page](https://www.flarion.io/pricing), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Third-party benchmark or review of Flarion, and named customer results (searched: flarion.io, flarion.io/benchmarks, flarion.io/about-us) - Funding amounts and round dates (searched: flarion.io/about-us, flarion.io) - Explicit list of supported Spark versions (searched: flarion.io/product, flarion.io/faqs, flarion.io/benchmarks) ## Flarion side by side with each other option - [Flarion and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-flarion/) - [Flarion and Apache Auron](/compare/apache-auron-vs-flarion/) - [Flarion and Apache DataFusion Comet](/compare/datafusion-comet-vs-flarion/) - [Flarion and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-flarion/) - [Flarion and Databricks Photon](/compare/databricks-photon-vs-flarion/) - [Flarion and DataPelago](/compare/datapelago-vs-flarion/) - [Flarion and DualBird](/compare/dualbird-vs-flarion/) - [Flarion and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-flarion/) - [Flarion and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-flarion/) - [Flarion and NVIDIA cuDF plugin for Apache Spark](/compare/flarion-vs-nvidia-rapids-spark/) - [Flarion and Speedata](/compare/flarion-vs-speedata/) ## Sources - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Flarion: flarion.io/about-us: https://www.flarion.io/about-us (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/flarion/ --- # Google Cloud Lightning Engine, Premium Spark engine on Google Cloud: the published facts Option profile, read 10 Nov 2026 **The short answer.** Lightning Engine is Google Cloud's accelerated execution environment in Managed Service for Apache Spark, announced in preview on 2025-05-30. Google's launch blog reports 3.6x on TPC-H-like queries at 10 TB versus open source Spark on similar infrastructure. In us-central1, serverless premium costs $0.089 per DCU-hour against $0.06 for standard. - **Kind:** Premium Spark engine on Google Cloud - **Website:** [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine) - **Name checked:** Lightning Engine still exists, but the host service is now named 'Managed Service for Apache Spark' (Dataproc and Serverless for Apache Spark docs now redirect to docs.cloud.google.com/managed-spark). Serverless premium tier is 'Powered by Lightning Engine'. Native Query Execution (NQE, Gluten and Velox) is now an optional component of Lightning Engine, enabled on clusters. - **Read:** 10 Nov 2026 ## What it does well - Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. - Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. ## Where it is limited - Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. - Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 | | The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | | The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Lightning Engine is a high-performance execution environment for Apache Spark on Managed Service for Apache Spark that accelerates compute-intensive workloads without requiring changes to Spark code. It targets DataFrame, Dataset and Spark SQL workloads; I/O-bound or RDD-heavy jobs may suit the standard engine. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | Google announced Lightning Engine in preview on May 30, 2025. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 | ## Ledger entries for Google Cloud Lightning Engine | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2025-05-30 | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No | | Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) | Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name | Google Cloud (its maker) | No | | Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) | Google Cloud's wording: 2.5x to 4x speedup over open source Spark | Google Cloud (its maker) | No | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No Spark code change stated | Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | Cluster | Image 2.3.3+ or 3.0.2+, engine flag | Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | Cloud | Google Cloud only | Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | | Vendor | Premium tier or cluster add-on | Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Independent third-party benchmark of Lightning Engine (searched: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, docs.cloud.google.com/managed-spark/pricing) ## Google Cloud Lightning Engine side by side with each other option - [Google Cloud Lightning Engine and Amazon EMR runtime for Apache Spark](/compare/dataproc-lightning-engine-vs-emr-spark-runtime/) - [Google Cloud Lightning Engine and Apache Auron](/compare/apache-auron-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Apache DataFusion Comet](/compare/datafusion-comet-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Databricks Photon](/compare/databricks-photon-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and DataPelago](/compare/datapelago-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and DualBird](/compare/dataproc-lightning-engine-vs-dualbird/) - [Google Cloud Lightning Engine and Flarion](/compare/dataproc-lightning-engine-vs-flarion/) - [Google Cloud Lightning Engine and Microsoft Fabric native execution engine](/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/) - [Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark](/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/) - [Google Cloud Lightning Engine and Speedata](/compare/dataproc-lightning-engine-vs-speedata/) ## Sources - Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Google Cloud Lightning Engine: cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/dataproc-lightning-engine/ --- # Microsoft Fabric native execution engine, Velox-based engine inside Fabric Spark: the published facts Option profile, read 10 Nov 2026 **The short answer.** The native execution engine in Microsoft Fabric Spark is based on Velox and Apache Gluten. Microsoft's documentation covers Runtime 1.3 (Spark 3.5) and Runtime 2.0 (Spark 4.1) and reading Parquet, Delta and CSV. It is enabled by an environment toggle or by setting spark.native.enabled to true. - **Kind:** Velox-based engine inside Fabric Spark - **Website:** [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview) - **Name checked:** Still documented as 'Native execution engine for Fabric Data Engineering'. The page still refers to 'the preview phase' in the enablement section, though it covers Runtime 1.3 and 2.0. - **Read:** 10 Nov 2026 ## What it does well - Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. - The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. ## Where it is limited - The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. - Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | vendor docs | [azure.microsoft.com/en-us/pricing/details/microsoft-fabric](https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | The engine is based on two open source components: Velox, a C++ library introduced by Meta, and Apache Gluten (incubating), a middle layer introduced by Intel that offloads JVM SQL engine execution to native engines. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | ## Ledger entries for Microsoft Fabric native execution engine None found on 10 Nov 2026. ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code modifications stated | Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | Cluster | Toggle in environment or Spark config | Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | Cloud | Microsoft Fabric on Azure only | The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | Vendor | Fabric capacity required | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | [Microsoft Fabric pricing](https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Published benchmark with workload, data size and result for the Fabric native execution engine (searched: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin, azure.microsoft.com/en-us/pricing/details/microsoft-fabric) - Public benchmark repo or scripts (searched: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin) ## Microsoft Fabric native execution engine side by side with each other option - [Microsoft Fabric native execution engine and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache Auron](/compare/apache-auron-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache DataFusion Comet](/compare/datafusion-comet-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Databricks Photon](/compare/databricks-photon-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and DataPelago](/compare/datapelago-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and DualBird](/compare/dualbird-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Flarion](/compare/fabric-native-execution-engine-vs-flarion/) - [Microsoft Fabric native execution engine and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark](/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/) - [Microsoft Fabric native execution engine and Speedata](/compare/fabric-native-execution-engine-vs-speedata/) ## Sources - Microsoft Fabric native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/fabric-native-execution-engine/ --- # NVIDIA cuDF plugin for Apache Spark, GPU plugin, formerly RAPIDS Accelerator: the published facts Option profile, read 10 Nov 2026 **The short answer.** The NVIDIA cuDF plugin for Apache Spark, formerly the RAPIDS Accelerator, runs supported Spark SQL and DataFrame operations on NVIDIA GPUs. AWS and NVIDIA report TPC-DS at 3 TB on eight g7.4xlarge nodes in 281 seconds, at $2.06 per run. The latest release, v26.10.0, is dated 2026-11-05. - **Kind:** GPU plugin, formerly RAPIDS Accelerator - **Website:** [docs.nvidia.com/spark-rapids/user-guide/latest/index.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html) - **Name checked:** The product still exists but is now documented as the NVIDIA cuDF plugin for Apache Spark. The GitHub repo NVIDIA/spark-rapids now resolves to NVIDIA/cudf-spark and the user guide is titled 'cuDF plugin for Apache Spark - User Guide'. Some artifacts keep the old name, for example the Amazon EMR image variant 'emr-7.12.0-spark-rapids-latest' and the config prefix spark.rapids. - **Read:** 10 Nov 2026 ## What it does well - AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. - Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. ## Where it is limited - The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. - Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. | cloud provider docs | [aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. | vendor docs | [docs.nvidia.com/spark-rapids/user-guide/latest/faq.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html), 10 Nov 2026 | | NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | vendor docs | [docs.nvidia.com/spark-rapids/user-guide/latest/overview.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | cloud provider docs | [aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. | vendor docs | [docs.nvidia.com/spark-rapids/user-guide/latest/overview.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html), 10 Nov 2026 | | Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | vendor docs | [docs.nvidia.com/spark-rapids/user-guide/latest/faq.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. | vendor docs | [docs.nvidia.com/spark-rapids/user-guide/latest/index.html](https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html), 10 Nov 2026 | | AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | cloud provider docs | [aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. | vendor docs | [nvidia.github.io/cudf-spark/docs/download.html](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | | Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | cloud provider docs | [aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | vendor docs | [nvidia.github.io/cudf-spark/docs/download.html](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | vendor | [raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md](https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | Owner is NVIDIA (copyright 2026 NVIDIA Corporation & Affiliates). The repo LICENSE says source code is Apache-2.0 and documentation is CC-BY-4.0; the GitHub API reports the license as 'other'. Repo created 2020-05-14, about 1,000 stars. | vendor | [raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE](https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Recent GitHub releases: v26.10.0 (2026-11-05), v26.08.1 (2026-08-28), v26.08.0 (2026-08-20). Date-based versioning with releases roughly every two months. | vendor | [api.github.com/repos/NVIDIA/cudf-spark/releases?per_page=3](https://api.github.com/repos/NVIDIA/cudf-spark/releases?per_page=3), 10 Nov 2026 | ## Ledger entries for NVIDIA cuDF plugin for Apache Spark | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-08-25 | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) | | Not stated | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) | NVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as published | NVIDIA (its maker) | No (the NDS automation is public; the comparison is described on the overview page only) | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code change stated | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | [NVIDIA Spark plugin overview](https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html), 10 Nov 2026 | | Cluster | GPU instances and plugin settings | The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | Cloud | AWS, Azure and Google Cloud documented | NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | [NVIDIA cuDF plugin download page](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | | Vendor | Free software; GPU instances paid | The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - An independent (non-NVIDIA, non-AWS) benchmark of the plugin on TPC-DS or ETL (searched: docs.nvidia.com/spark-rapids/user-guide/latest/overview.html, nvidia.github.io/cudf-spark, aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e) ## NVIDIA cuDF plugin for Apache Spark side by side with each other option - [NVIDIA cuDF plugin for Apache Spark and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache Auron](/compare/apache-auron-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache DataFusion Comet](/compare/datafusion-comet-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Databricks Photon](/compare/databricks-photon-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and DataPelago](/compare/datapelago-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and DualBird](/compare/dualbird-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Flarion](/compare/flarion-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Speedata](/compare/nvidia-rapids-spark-vs-speedata/) ## Sources - NVIDIA cuDF plugin for Apache Spark: aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE: https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: api.github.com/repos/NVIDIA/cudf-spark/releases?per_page=3: https://api.github.com/repos/NVIDIA/cudf-spark/releases?per_page=3 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/nvidia-rapids-spark/ --- # Speedata, PCIe analytics chip plus Dash Spark plugin: the published facts Option profile, read 10 Nov 2026 **The short answer.** Speedata makes the C200 Analytics Processing Unit, a PCIe Gen 5 card with the Callisto ASIC, and the Dash plugin that offloads Spark SQL work to it. Speedata claims up to 100x acceleration and up to 90% lower TCO. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or a Speedata-hosted AWS service. - **Kind:** PCIe analytics chip plus Dash Spark plugin - **Website:** [www.speedata.io](https://www.speedata.io/) - **Name checked:** speedata.io is live and redirects to www.speedata.io under the name Speedata. Current CEO is Adi Gelvan, who joined in 2025. - **Read:** 10 Nov 2026 ## What it does well - Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. - Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. ## Where it is limited - The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. - No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. | vendor | [speedata.io](https://www.speedata.io/), 10 Nov 2026 | | Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. | vendor | [speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-](https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu), 10 Nov 2026 | | The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | vendor | [speedata.io/product](https://www.speedata.io/product), 10 Nov 2026 | ## Published cost results and prices | Fact | Source type | Source, read | | --- | --- | --- | | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | vendor | [speedata.io/product](https://www.speedata.io/product), 10 Nov 2026 | ## Code changes | Fact | Source type | Source, read | | --- | --- | --- | | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | vendor | [speedata.io/product](https://www.speedata.io/product), 10 Nov 2026 | ## Tuning | Fact | Source type | Source, read | | --- | --- | --- | | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | vendor | [speedata.io/workload-analyzer](https://www.speedata.io/workload-analyzer), 10 Nov 2026 | ## Spark versions, platforms and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | vendor | [speedata.io/technology](https://www.speedata.io/technology), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | vendor | [speedata.io](https://www.speedata.io/), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | vendor | [github.com/Speedata-io](https://github.com/Speedata-io), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Company page: founded by semiconductor veterans, including Jonathan Friedmann (Provigent, acquired by Broadcom for over $300M) and Yoav Etsion (Technion professor). CEO Adi Gelvan (co-founder of Speedb, acquired by Redis in 2024) joined in 2025 after a successful chip tape-out. Press list includes a SemiWiki CEO interview (May 17, 2026) and a PR Newswire release on the Nebul partnership (Feb 19, 2026). Partners listed: Dell, HPE, Nebul, Carahsoft, Systemic.fr, Kodata. | vendor | [speedata.io/company](https://www.speedata.io/company), 10 Nov 2026 | ## Ledger entries for Speedata | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-06-11 | [Speedata](/options/speedata/) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched | | Not stated | [Speedata](/options/speedata/) | [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) | Speedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing time | Speedata (its maker) | No | ## Change Budget row | Column | What changes | Detail | Source, read | | --- | --- | --- | --- | | Code | No code change stated | Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | | Cluster | PCIe C200 card in each server | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | | Cloud | On-premises, OEM or hosted AWS | Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 | | Vendor | Speedata purchase; no price list | No price list is published. The pages offer buying a server or going through OEM or cloud partners. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | ## Searched for and none found on 10 Nov 2026 - Independent Spark benchmark (TPC-DS or ETL) of the APU with published method, and the Microsoft paper itself (searched: speedata.io/news, speedata.io/blog, github.com/Speedata-io) - Funding amounts and investors (searched: speedata.io/company, speedata.io/news, speedata.io) - EMR, EKS, Databricks and table format support (searched: speedata.io/product, speedata.io/technology, speedata.io) ## Speedata side by side with each other option - [Speedata and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-speedata/) - [Speedata and Apache Auron](/compare/apache-auron-vs-speedata/) - [Speedata and Apache DataFusion Comet](/compare/datafusion-comet-vs-speedata/) - [Speedata and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-speedata/) - [Speedata and Databricks Photon](/compare/databricks-photon-vs-speedata/) - [Speedata and DataPelago](/compare/datapelago-vs-speedata/) - [Speedata and DualBird](/compare/dualbird-vs-speedata/) - [Speedata and Flarion](/compare/flarion-vs-speedata/) - [Speedata and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-speedata/) - [Speedata and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-speedata/) - [Speedata and NVIDIA cuDF plugin for Apache Spark](/compare/nvidia-rapids-spark-vs-speedata/) ## Sources - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) - Speedata: speedata.io/company: https://www.speedata.io/company (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/speedata/ --- # 66 pairs of options, side by side in the same rows Side by side, read 10 Nov 2026 **The short answer.** Pick any two of the twelve options and read their sourced facts side by side: where each runs, what it changes, how it is licensed or priced, which Spark versions it lists and which benchmarks exist. Pairs are set in alphabetical order, and each comparison ends with facts, not with a conclusion. ## Amazon EMR runtime for Apache Spark - [Amazon EMR runtime for Apache Spark and Apache Auron](/compare/apache-auron-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Apache DataFusion Comet](/compare/datafusion-comet-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Databricks Photon](/compare/databricks-photon-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and DataPelago](/compare/datapelago-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and DualBird](/compare/dualbird-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Flarion](/compare/emr-spark-runtime-vs-flarion/) - [Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-emr-spark-runtime/) - [Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine](/compare/emr-spark-runtime-vs-fabric-native-execution-engine/) - [Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark](/compare/emr-spark-runtime-vs-nvidia-rapids-spark/) - [Amazon EMR runtime for Apache Spark and Speedata](/compare/emr-spark-runtime-vs-speedata/) ## Apache Auron - [Apache Auron and Amazon EMR runtime for Apache Spark](/compare/apache-auron-vs-emr-spark-runtime/) - [Apache Auron and Apache DataFusion Comet](/compare/apache-auron-vs-datafusion-comet/) - [Apache Auron and Apache Gluten with Velox](/compare/apache-auron-vs-apache-gluten-velox/) - [Apache Auron and Databricks Photon](/compare/apache-auron-vs-databricks-photon/) - [Apache Auron and DataPelago](/compare/apache-auron-vs-datapelago/) - [Apache Auron and DualBird](/compare/apache-auron-vs-dualbird/) - [Apache Auron and Flarion](/compare/apache-auron-vs-flarion/) - [Apache Auron and Google Cloud Lightning Engine](/compare/apache-auron-vs-dataproc-lightning-engine/) - [Apache Auron and Microsoft Fabric native execution engine](/compare/apache-auron-vs-fabric-native-execution-engine/) - [Apache Auron and NVIDIA cuDF plugin for Apache Spark](/compare/apache-auron-vs-nvidia-rapids-spark/) - [Apache Auron and Speedata](/compare/apache-auron-vs-speedata/) ## Apache DataFusion Comet - [Apache DataFusion Comet and Amazon EMR runtime for Apache Spark](/compare/datafusion-comet-vs-emr-spark-runtime/) - [Apache DataFusion Comet and Apache Auron](/compare/apache-auron-vs-datafusion-comet/) - [Apache DataFusion Comet and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-datafusion-comet/) - [Apache DataFusion Comet and Databricks Photon](/compare/databricks-photon-vs-datafusion-comet/) - [Apache DataFusion Comet and DataPelago](/compare/datafusion-comet-vs-datapelago/) - [Apache DataFusion Comet and DualBird](/compare/datafusion-comet-vs-dualbird/) - [Apache DataFusion Comet and Flarion](/compare/datafusion-comet-vs-flarion/) - [Apache DataFusion Comet and Google Cloud Lightning Engine](/compare/datafusion-comet-vs-dataproc-lightning-engine/) - [Apache DataFusion Comet and Microsoft Fabric native execution engine](/compare/datafusion-comet-vs-fabric-native-execution-engine/) - [Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark](/compare/datafusion-comet-vs-nvidia-rapids-spark/) - [Apache DataFusion Comet and Speedata](/compare/datafusion-comet-vs-speedata/) ## Apache Gluten with Velox - [Apache Gluten with Velox and Amazon EMR runtime for Apache Spark](/compare/apache-gluten-velox-vs-emr-spark-runtime/) - [Apache Gluten with Velox and Apache Auron](/compare/apache-auron-vs-apache-gluten-velox/) - [Apache Gluten with Velox and Apache DataFusion Comet](/compare/apache-gluten-velox-vs-datafusion-comet/) - [Apache Gluten with Velox and Databricks Photon](/compare/apache-gluten-velox-vs-databricks-photon/) - [Apache Gluten with Velox and DataPelago](/compare/apache-gluten-velox-vs-datapelago/) - [Apache Gluten with Velox and DualBird](/compare/apache-gluten-velox-vs-dualbird/) - [Apache Gluten with Velox and Flarion](/compare/apache-gluten-velox-vs-flarion/) - [Apache Gluten with Velox and Google Cloud Lightning Engine](/compare/apache-gluten-velox-vs-dataproc-lightning-engine/) - [Apache Gluten with Velox and Microsoft Fabric native execution engine](/compare/apache-gluten-velox-vs-fabric-native-execution-engine/) - [Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark](/compare/apache-gluten-velox-vs-nvidia-rapids-spark/) - [Apache Gluten with Velox and Speedata](/compare/apache-gluten-velox-vs-speedata/) ## Databricks Photon - [Databricks Photon and Amazon EMR runtime for Apache Spark](/compare/databricks-photon-vs-emr-spark-runtime/) - [Databricks Photon and Apache Auron](/compare/apache-auron-vs-databricks-photon/) - [Databricks Photon and Apache DataFusion Comet](/compare/databricks-photon-vs-datafusion-comet/) - [Databricks Photon and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-databricks-photon/) - [Databricks Photon and DataPelago](/compare/databricks-photon-vs-datapelago/) - [Databricks Photon and DualBird](/compare/databricks-photon-vs-dualbird/) - [Databricks Photon and Flarion](/compare/databricks-photon-vs-flarion/) - [Databricks Photon and Google Cloud Lightning Engine](/compare/databricks-photon-vs-dataproc-lightning-engine/) - [Databricks Photon and Microsoft Fabric native execution engine](/compare/databricks-photon-vs-fabric-native-execution-engine/) - [Databricks Photon and NVIDIA cuDF plugin for Apache Spark](/compare/databricks-photon-vs-nvidia-rapids-spark/) - [Databricks Photon and Speedata](/compare/databricks-photon-vs-speedata/) ## DataPelago - [DataPelago and Amazon EMR runtime for Apache Spark](/compare/datapelago-vs-emr-spark-runtime/) - [DataPelago and Apache Auron](/compare/apache-auron-vs-datapelago/) - [DataPelago and Apache DataFusion Comet](/compare/datafusion-comet-vs-datapelago/) - [DataPelago and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-datapelago/) - [DataPelago and Databricks Photon](/compare/databricks-photon-vs-datapelago/) - [DataPelago and DualBird](/compare/datapelago-vs-dualbird/) - [DataPelago and Flarion](/compare/datapelago-vs-flarion/) - [DataPelago and Google Cloud Lightning Engine](/compare/datapelago-vs-dataproc-lightning-engine/) - [DataPelago and Microsoft Fabric native execution engine](/compare/datapelago-vs-fabric-native-execution-engine/) - [DataPelago and NVIDIA cuDF plugin for Apache Spark](/compare/datapelago-vs-nvidia-rapids-spark/) - [DataPelago and Speedata](/compare/datapelago-vs-speedata/) ## DualBird - [DualBird and Amazon EMR runtime for Apache Spark](/compare/dualbird-vs-emr-spark-runtime/) - [DualBird and Apache Auron](/compare/apache-auron-vs-dualbird/) - [DualBird and Apache DataFusion Comet](/compare/datafusion-comet-vs-dualbird/) - [DualBird and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-dualbird/) - [DualBird and Databricks Photon](/compare/databricks-photon-vs-dualbird/) - [DualBird and DataPelago](/compare/datapelago-vs-dualbird/) - [DualBird and Flarion](/compare/dualbird-vs-flarion/) - [DualBird and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-dualbird/) - [DualBird and Microsoft Fabric native execution engine](/compare/dualbird-vs-fabric-native-execution-engine/) - [DualBird and NVIDIA cuDF plugin for Apache Spark](/compare/dualbird-vs-nvidia-rapids-spark/) - [DualBird and Speedata](/compare/dualbird-vs-speedata/) ## Flarion - [Flarion and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-flarion/) - [Flarion and Apache Auron](/compare/apache-auron-vs-flarion/) - [Flarion and Apache DataFusion Comet](/compare/datafusion-comet-vs-flarion/) - [Flarion and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-flarion/) - [Flarion and Databricks Photon](/compare/databricks-photon-vs-flarion/) - [Flarion and DataPelago](/compare/datapelago-vs-flarion/) - [Flarion and DualBird](/compare/dualbird-vs-flarion/) - [Flarion and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-flarion/) - [Flarion and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-flarion/) - [Flarion and NVIDIA cuDF plugin for Apache Spark](/compare/flarion-vs-nvidia-rapids-spark/) - [Flarion and Speedata](/compare/flarion-vs-speedata/) ## Google Cloud Lightning Engine - [Google Cloud Lightning Engine and Amazon EMR runtime for Apache Spark](/compare/dataproc-lightning-engine-vs-emr-spark-runtime/) - [Google Cloud Lightning Engine and Apache Auron](/compare/apache-auron-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Apache DataFusion Comet](/compare/datafusion-comet-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and Databricks Photon](/compare/databricks-photon-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and DataPelago](/compare/datapelago-vs-dataproc-lightning-engine/) - [Google Cloud Lightning Engine and DualBird](/compare/dataproc-lightning-engine-vs-dualbird/) - [Google Cloud Lightning Engine and Flarion](/compare/dataproc-lightning-engine-vs-flarion/) - [Google Cloud Lightning Engine and Microsoft Fabric native execution engine](/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/) - [Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark](/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/) - [Google Cloud Lightning Engine and Speedata](/compare/dataproc-lightning-engine-vs-speedata/) ## Microsoft Fabric native execution engine - [Microsoft Fabric native execution engine and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache Auron](/compare/apache-auron-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache DataFusion Comet](/compare/datafusion-comet-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Databricks Photon](/compare/databricks-photon-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and DataPelago](/compare/datapelago-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and DualBird](/compare/dualbird-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and Flarion](/compare/fabric-native-execution-engine-vs-flarion/) - [Microsoft Fabric native execution engine and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/) - [Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark](/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/) - [Microsoft Fabric native execution engine and Speedata](/compare/fabric-native-execution-engine-vs-speedata/) ## NVIDIA cuDF plugin for Apache Spark - [NVIDIA cuDF plugin for Apache Spark and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache Auron](/compare/apache-auron-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache DataFusion Comet](/compare/datafusion-comet-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Databricks Photon](/compare/databricks-photon-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and DataPelago](/compare/datapelago-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and DualBird](/compare/dualbird-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Flarion](/compare/flarion-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/) - [NVIDIA cuDF plugin for Apache Spark and Speedata](/compare/nvidia-rapids-spark-vs-speedata/) ## Speedata - [Speedata and Amazon EMR runtime for Apache Spark](/compare/emr-spark-runtime-vs-speedata/) - [Speedata and Apache Auron](/compare/apache-auron-vs-speedata/) - [Speedata and Apache DataFusion Comet](/compare/datafusion-comet-vs-speedata/) - [Speedata and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-speedata/) - [Speedata and Databricks Photon](/compare/databricks-photon-vs-speedata/) - [Speedata and DataPelago](/compare/datapelago-vs-speedata/) - [Speedata and DualBird](/compare/dualbird-vs-speedata/) - [Speedata and Flarion](/compare/flarion-vs-speedata/) - [Speedata and Google Cloud Lightning Engine](/compare/dataproc-lightning-engine-vs-speedata/) - [Speedata and Microsoft Fabric native execution engine](/compare/fabric-native-execution-engine-vs-speedata/) - [Speedata and NVIDIA cuDF plugin for Apache Spark](/compare/nvidia-rapids-spark-vs-speedata/) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/ --- # Amazon EMR runtime for Apache Spark and Apache Auron: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Apache Auron are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 1 entry for Apache Auron. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Apache Auron | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Open-source Rust engine plugin for Spark | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | None found on 10 Nov 2026. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and Apache DataFusion Comet: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Apache DataFusion Comet are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 2 entries for Apache DataFusion Comet. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Apache DataFusion Comet | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Open-source Spark plugin on DataFusion | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | None found on 10 Nov 2026. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and Apache Gluten with Velox: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Apache Gluten with Velox are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 2 entries for Apache Gluten with Velox. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Apache Gluten with Velox | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Open-source Spark plugin offloading to Velox | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | None found on 10 Nov 2026. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and Databricks Photon: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Databricks Photon are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 2 entries for Databricks Photon. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Databricks Photon | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Databricks's C++ vectorized engine | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and DataPelago: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and DataPelago are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Amazon EMR runtime for Apache Spark and 1 entry for DataPelago, each with who ran it. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | DataPelago | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Gluten-based CPU and GPU accelerator | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | None found on 10 Nov 2026. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and DualBird are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Amazon EMR runtime for Apache Spark and 3 entries for DualBird, each with who ran it. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | DualBird | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | FPGA Spark plugin on AWS F2 | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dualbird-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Flarion are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Amazon EMR runtime for Apache Spark and 2 entries for Flarion, each with who ran it. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Flarion | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Rust and DataFusion engine as one JAR | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-flarion/ --- # Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 3 entries for Google Cloud Lightning Engine. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Premium Spark engine on Google Cloud | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-emr-spark-runtime/ --- # Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and no entries for Microsoft Fabric native execution engine. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | Velox-based engine inside Fabric Spark | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-fabric-native-execution-engine/ --- # Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | GPU plugin, formerly RAPIDS Accelerator | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-nvidia-rapids-spark/ --- # Amazon EMR runtime for Apache Spark and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR runtime for Apache Spark and Speedata are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Amazon EMR runtime for Apache Spark and 2 entries for Speedata, each with who ran it. ## The same rows for both | Row | Amazon EMR runtime for Apache Spark | Speedata | | --- | --- | --- | | Kind | AWS-tuned Spark runtime on EMR | PCIe analytics chip plus Dash Spark plugin | | Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/emr-spark-runtime-vs-speedata/ --- # Apache Auron and Apache DataFusion Comet: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron and Apache DataFusion Comet are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Apache Auron and 2 entries for Apache DataFusion Comet, each with who ran it. ## The same rows for both | Row | Apache Auron | Apache DataFusion Comet | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Open-source Spark plugin on DataFusion | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | | Published cost results and prices | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-datafusion-comet/ --- # Apache Auron and Apache Gluten with Velox: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron and Apache Gluten with Velox are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Apache Auron and 2 entries for Apache Gluten with Velox, each with who ran it. ## The same rows for both | Row | Apache Auron | Apache Gluten with Velox | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Open-source Spark plugin offloading to Velox | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | | Published cost results and prices | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-apache-gluten-velox/ --- # Apache Auron and Databricks Photon: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron (Open-source Rust engine plugin for Spark) and Databricks Photon (Databricks's C++ vectorized engine) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for Apache Auron and 2 entries for Databricks Photon. ## The same rows for both | Row | Apache Auron | Databricks Photon | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Databricks's C++ vectorized engine | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | | Published cost results and prices | None found on 10 Nov 2026. | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-databricks-photon/ --- # Apache Auron and DataPelago: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron (Open-source Rust engine plugin for Spark) and DataPelago (Gluten-based CPU and GPU accelerator) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for Apache Auron and 1 entry for DataPelago. ## The same rows for both | Row | Apache Auron | DataPelago | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Gluten-based CPU and GPU accelerator | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | | Published cost results and prices | None found on 10 Nov 2026. | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | None found on 10 Nov 2026. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-datapelago/ --- # Apache Auron and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron (Open-source Rust engine plugin for Spark) and DualBird (FPGA Spark plugin on AWS F2) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for Apache Auron and 3 entries for DualBird. ## The same rows for both | Row | Apache Auron | DualBird | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | FPGA Spark plugin on AWS F2 | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | None found on 10 Nov 2026. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-dualbird/ --- # Apache Auron and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron (Open-source Rust engine plugin for Spark) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for Apache Auron and 2 entries for Flarion. ## The same rows for both | Row | Apache Auron | Flarion | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Rust and DataFusion engine as one JAR | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | None found on 10 Nov 2026. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-flarion/ --- # Apache Auron and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Apache Auron and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | Apache Auron | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Premium Spark engine on Google Cloud | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | None found on 10 Nov 2026. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-dataproc-lightning-engine/ --- # Apache Auron and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for Apache Auron and no entries for Microsoft Fabric native execution engine, each with who ran it. ## The same rows for both | Row | Apache Auron | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | Velox-based engine inside Fabric Spark | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | None found on 10 Nov 2026. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-fabric-native-execution-engine/ --- # Apache Auron and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Apache Auron and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Apache Auron | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | GPU plugin, formerly RAPIDS Accelerator | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | None found on 10 Nov 2026. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-nvidia-rapids-spark/ --- # Apache Auron and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Auron (Open-source Rust engine plugin for Spark) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for Apache Auron and 2 entries for Speedata. ## The same rows for both | Row | Apache Auron | Speedata | | --- | --- | --- | | Kind | Open-source Rust engine plugin for Spark | PCIe analytics chip plus Dash Spark plugin | | Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | None found on 10 Nov 2026. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Auron (incubating), formerly Blaze: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-auron-vs-speedata/ --- # Apache DataFusion Comet and Apache Gluten with Velox: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet and Apache Gluten with Velox are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache DataFusion Comet and 2 entries for Apache Gluten with Velox, each with who ran it. ## The same rows for both | Row | Apache DataFusion Comet | Apache Gluten with Velox | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Open-source Spark plugin offloading to Velox | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | | Published cost results and prices | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-datafusion-comet/ --- # Apache DataFusion Comet and Databricks Photon: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and Databricks Photon (Databricks's C++ vectorized engine) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for Databricks Photon. ## The same rows for both | Row | Apache DataFusion Comet | Databricks Photon | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Databricks's C++ vectorized engine | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | | Published cost results and prices | None found on 10 Nov 2026. | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-datafusion-comet/ --- # Apache DataFusion Comet and DataPelago: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and DataPelago (Gluten-based CPU and GPU accelerator) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 1 entry for DataPelago. ## The same rows for both | Row | Apache DataFusion Comet | DataPelago | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Gluten-based CPU and GPU accelerator | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | | Published cost results and prices | None found on 10 Nov 2026. | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | None found on 10 Nov 2026. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-datapelago/ --- # Apache DataFusion Comet and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and DualBird (FPGA Spark plugin on AWS F2) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 3 entries for DualBird. ## The same rows for both | Row | Apache DataFusion Comet | DualBird | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | FPGA Spark plugin on AWS F2 | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | None found on 10 Nov 2026. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-dualbird/ --- # Apache DataFusion Comet and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for Flarion. ## The same rows for both | Row | Apache DataFusion Comet | Flarion | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Rust and DataFusion engine as one JAR | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | None found on 10 Nov 2026. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-flarion/ --- # Apache DataFusion Comet and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache DataFusion Comet and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | Apache DataFusion Comet | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Premium Spark engine on Google Cloud | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | None found on 10 Nov 2026. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-dataproc-lightning-engine/ --- # Apache DataFusion Comet and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache DataFusion Comet and no entries for Microsoft Fabric native execution engine. ## The same rows for both | Row | Apache DataFusion Comet | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | Velox-based engine inside Fabric Spark | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | None found on 10 Nov 2026. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-fabric-native-execution-engine/ --- # Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Apache DataFusion Comet | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | GPU plugin, formerly RAPIDS Accelerator | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | None found on 10 Nov 2026. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-nvidia-rapids-spark/ --- # Apache DataFusion Comet and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for Speedata. ## The same rows for both | Row | Apache DataFusion Comet | Speedata | | --- | --- | --- | | Kind | Open-source Spark plugin on DataFusion | PCIe analytics chip plus Dash Spark plugin | | Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | None found on 10 Nov 2026. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-speedata/ --- # Apache Gluten with Velox and Databricks Photon: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and Databricks Photon are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache Gluten with Velox and 2 entries for Databricks Photon, each with who ran it. ## The same rows for both | Row | Apache Gluten with Velox | Databricks Photon | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | Databricks's C++ vectorized engine | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | | Published cost results and prices | None found on 10 Nov 2026. | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-databricks-photon/ --- # Apache Gluten with Velox and DataPelago: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and DataPelago are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache Gluten with Velox and 1 entry for DataPelago, each with who ran it. ## The same rows for both | Row | Apache Gluten with Velox | DataPelago | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | Gluten-based CPU and GPU accelerator | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | | Published cost results and prices | None found on 10 Nov 2026. | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | None found on 10 Nov 2026. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-datapelago/ --- # Apache Gluten with Velox and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and DualBird are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache Gluten with Velox and 3 entries for DualBird, each with who ran it. ## The same rows for both | Row | Apache Gluten with Velox | DualBird | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | FPGA Spark plugin on AWS F2 | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | None found on 10 Nov 2026. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-dualbird/ --- # Apache Gluten with Velox and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and Flarion are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache Gluten with Velox and 2 entries for Flarion, each with who ran it. ## The same rows for both | Row | Apache Gluten with Velox | Flarion | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | Rust and DataFusion engine as one JAR | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | None found on 10 Nov 2026. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-flarion/ --- # Apache Gluten with Velox and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache Gluten with Velox and 3 entries for Google Cloud Lightning Engine. ## The same rows for both | Row | Apache Gluten with Velox | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | Premium Spark engine on Google Cloud | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | None found on 10 Nov 2026. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-dataproc-lightning-engine/ --- # Apache Gluten with Velox and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache Gluten with Velox and no entries for Microsoft Fabric native execution engine. ## The same rows for both | Row | Apache Gluten with Velox | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | Velox-based engine inside Fabric Spark | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | None found on 10 Nov 2026. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-fabric-native-execution-engine/ --- # Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache Gluten with Velox and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Apache Gluten with Velox | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | GPU plugin, formerly RAPIDS Accelerator | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | None found on 10 Nov 2026. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-nvidia-rapids-spark/ --- # Apache Gluten with Velox and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Gluten with Velox and Speedata are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache Gluten with Velox and 2 entries for Speedata, each with who ran it. ## The same rows for both | Row | Apache Gluten with Velox | Speedata | | --- | --- | --- | | Kind | Open-source Spark plugin offloading to Velox | PCIe analytics chip plus Dash Spark plugin | | Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | None found on 10 Nov 2026. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) (run by its maker); [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Apache Gluten with the Velox backend: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - Apache Gluten with the Velox backend: gluten.apache.org/downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/apache-gluten-velox-vs-speedata/ --- # Databricks Photon and DataPelago: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon (Databricks's C++ vectorized engine) and DataPelago (Gluten-based CPU and GPU accelerator) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Databricks Photon and 1 entry for DataPelago. ## The same rows for both | Row | Databricks Photon | DataPelago | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | Gluten-based CPU and GPU accelerator | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | None found on 10 Nov 2026. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-datapelago/ --- # Databricks Photon and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon (Databricks's C++ vectorized engine) and DualBird (FPGA Spark plugin on AWS F2) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Databricks Photon and 3 entries for DualBird. ## The same rows for both | Row | Databricks Photon | DualBird | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | FPGA Spark plugin on AWS F2 | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-dualbird/ --- # Databricks Photon and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon (Databricks's C++ vectorized engine) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Databricks Photon and 2 entries for Flarion. ## The same rows for both | Row | Databricks Photon | Flarion | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | Rust and DataFusion engine as one JAR | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-flarion/ --- # Databricks Photon and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Databricks Photon and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | Databricks Photon | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | Premium Spark engine on Google Cloud | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-dataproc-lightning-engine/ --- # Databricks Photon and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Databricks Photon and no entries for Microsoft Fabric native execution engine, each with who ran it. ## The same rows for both | Row | Databricks Photon | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | Velox-based engine inside Fabric Spark | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-fabric-native-execution-engine/ --- # Databricks Photon and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Databricks Photon and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Databricks Photon | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | GPU plugin, formerly RAPIDS Accelerator | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-nvidia-rapids-spark/ --- # Databricks Photon and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks Photon (Databricks's C++ vectorized engine) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Databricks Photon and 2 entries for Speedata. ## The same rows for both | Row | Databricks Photon | Speedata | | --- | --- | --- | | Kind | Databricks's C++ vectorized engine | PCIe analytics chip plus Dash Spark plugin | | Does well | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) (run by its maker); [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Databricks Photon: docs.databricks.com/en/compute/photon.html: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks Photon: databricks.com/product/pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/databricks-photon-vs-speedata/ --- # DataPelago and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago (Gluten-based CPU and GPU accelerator) and DualBird (FPGA Spark plugin on AWS F2) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for DataPelago and 3 entries for DualBird. ## The same rows for both | Row | DataPelago | DualBird | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | FPGA Spark plugin on AWS F2 | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | None found on 10 Nov 2026. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-dualbird/ --- # DataPelago and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago (Gluten-based CPU and GPU accelerator) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for DataPelago and 2 entries for Flarion. ## The same rows for both | Row | DataPelago | Flarion | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | Rust and DataFusion engine as one JAR | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | None found on 10 Nov 2026. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-flarion/ --- # DataPelago and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for DataPelago and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | DataPelago | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | Premium Spark engine on Google Cloud | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | None found on 10 Nov 2026. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-dataproc-lightning-engine/ --- # DataPelago and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for DataPelago and no entries for Microsoft Fabric native execution engine, each with who ran it. ## The same rows for both | Row | DataPelago | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | Velox-based engine inside Fabric Spark | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | None found on 10 Nov 2026. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-fabric-native-execution-engine/ --- # DataPelago and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 published entry for DataPelago and 2 entries for NVIDIA cuDF plugin for Apache Spark, each with who ran it. ## The same rows for both | Row | DataPelago | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | GPU plugin, formerly RAPIDS Accelerator | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | None found on 10 Nov 2026. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-nvidia-rapids-spark/ --- # DataPelago and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DataPelago (Gluten-based CPU and GPU accelerator) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 1 entry for DataPelago and 2 entries for Speedata. ## The same rows for both | Row | DataPelago | Speedata | | --- | --- | --- | | Kind | Gluten-based CPU and GPU accelerator | PCIe analytics chip plus Dash Spark plugin | | Does well | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | None found on 10 Nov 2026. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DataPelago: datapelago.ai/dpa: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DataPelago: datapelago.ai/technology: https://www.datapelago.ai/technology (read 10 Nov 2026) - DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datapelago-vs-speedata/ --- # DualBird and Flarion: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin on AWS F2) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 entries for DualBird and 2 entries for Flarion. ## The same rows for both | Row | DualBird | Flarion | | --- | --- | --- | | Kind | FPGA Spark plugin on AWS F2 | Rust and DataFusion engine as one JAR | | Does well | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | | Limited | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | | Published cost results and prices | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | | Code changes | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | | Tuning | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | | Spark versions, platforms and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | | Change Budget: code | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | | Change Budget: cluster | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | | Change Budget: cloud | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | | Change Budget: vendor | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | Ledger entries | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dualbird-vs-flarion/ --- # DualBird and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 published entries for DualBird and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | DualBird | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | FPGA Spark plugin on AWS F2 | Premium Spark engine on Google Cloud | | Does well | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-dualbird/ --- # DualBird and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 published entries for DualBird and no entries for Microsoft Fabric native execution engine, each with who ran it. ## The same rows for both | Row | DualBird | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | FPGA Spark plugin on AWS F2 | Velox-based engine inside Fabric Spark | | Does well | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dualbird-vs-fabric-native-execution-engine/ --- # DualBird and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 published entries for DualBird and 2 entries for NVIDIA cuDF plugin for Apache Spark, each with who ran it. ## The same rows for both | Row | DualBird | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | FPGA Spark plugin on AWS F2 | GPU plugin, formerly RAPIDS Accelerator | | Does well | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dualbird-vs-nvidia-rapids-spark/ --- # DualBird and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin on AWS F2) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 entries for DualBird and 2 entries for Speedata. ## The same rows for both | Row | DualBird | Speedata | | --- | --- | --- | | Kind | FPGA Spark plugin on AWS F2 | PCIe analytics chip plus Dash Spark plugin | | Does well | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) (run by its maker); [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) (run by its maker); [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dualbird-vs-speedata/ --- # Flarion and Google Cloud Lightning Engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Flarion and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Flarion and 3 entries for Google Cloud Lightning Engine, each with who ran it. ## The same rows for both | Row | Flarion | Google Cloud Lightning Engine | | --- | --- | --- | | Kind | Rust and DataFusion engine as one JAR | Premium Spark engine on Google Cloud | | Does well | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | | Limited | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | | Published speed results | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | | Published cost results and prices | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Code changes | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Tuning | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | | Spark versions, platforms and table formats | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | | Where it runs | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | | How open its benchmarks are | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | | Change Budget: code | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | | Change Budget: cluster | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | | Change Budget: cloud | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | | Change Budget: vendor | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | Ledger entries | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-flarion/ --- # Flarion and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Flarion and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Flarion and no entries for Microsoft Fabric native execution engine, each with who ran it. ## The same rows for both | Row | Flarion | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Rust and DataFusion engine as one JAR | Velox-based engine inside Fabric Spark | | Does well | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-flarion/ --- # Flarion and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Flarion and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Flarion and 2 entries for NVIDIA cuDF plugin for Apache Spark, each with who ran it. ## The same rows for both | Row | Flarion | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Rust and DataFusion engine as one JAR | GPU plugin, formerly RAPIDS Accelerator | | Does well | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/flarion-vs-nvidia-rapids-spark/ --- # Flarion and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Flarion (Rust and DataFusion engine as one JAR) and Speedata (PCIe analytics chip plus Dash Spark plugin) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Flarion and 2 entries for Speedata. ## The same rows for both | Row | Flarion | Speedata | | --- | --- | --- | | Kind | Rust and DataFusion engine as one JAR | PCIe analytics chip plus Dash Spark plugin | | Does well | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/flarion-vs-speedata/ --- # Google Cloud Lightning Engine and Microsoft Fabric native execution engine: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google Cloud Lightning Engine and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 3 entries for Google Cloud Lightning Engine and no entries for Microsoft Fabric native execution engine. ## The same rows for both | Row | Google Cloud Lightning Engine | Microsoft Fabric native execution engine | | --- | --- | --- | | Kind | Premium Spark engine on Google Cloud | Velox-based engine inside Fabric Spark | | Does well | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | | Limited | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | | Published speed results | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | | Published cost results and prices | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | | Code changes | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | | Tuning | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | | Spark versions, platforms and table formats | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | | Where it runs | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | | How open its benchmarks are | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | | Change Budget: code | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | | Change Budget: cluster | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | | Change Budget: cloud | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | | Change Budget: vendor | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | Ledger entries | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | None found on 10 Nov 2026. | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/ --- # Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 3 entries for Google Cloud Lightning Engine and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Google Cloud Lightning Engine | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Premium Spark engine on Google Cloud | GPU plugin, formerly RAPIDS Accelerator | | Does well | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/ --- # Google Cloud Lightning Engine and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google Cloud Lightning Engine and Speedata are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 3 published entries for Google Cloud Lightning Engine and 2 entries for Speedata, each with who ran it. ## The same rows for both | Row | Google Cloud Lightning Engine | Speedata | | --- | --- | --- | | Kind | Premium Spark engine on Google Cloud | PCIe analytics chip plus Dash Spark plugin | | Does well | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-speedata/ --- # Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric native execution engine and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds no entries for Microsoft Fabric native execution engine and 2 entries for NVIDIA cuDF plugin for Apache Spark. ## The same rows for both | Row | Microsoft Fabric native execution engine | NVIDIA cuDF plugin for Apache Spark | | --- | --- | --- | | Kind | Velox-based engine inside Fabric Spark | GPU plugin, formerly RAPIDS Accelerator | | Does well | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | | Limited | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | | Published speed results | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | | Published cost results and prices | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | | Code changes | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | | Tuning | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | | Spark versions, platforms and table formats | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | | Where it runs | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | | How open its benchmarks are | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | | Change Budget: code | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | | Change Budget: cluster | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | | Change Budget: cloud | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | | Change Budget: vendor | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | Ledger entries | None found on 10 Nov 2026. | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-nvidia-rapids-spark/ --- # Microsoft Fabric native execution engine and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric native execution engine and Speedata are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds no published entries for Microsoft Fabric native execution engine and 2 entries for Speedata, each with who ran it. ## The same rows for both | Row | Microsoft Fabric native execution engine | Speedata | | --- | --- | --- | | Kind | Velox-based engine inside Fabric Spark | PCIe analytics chip plus Dash Spark plugin | | Does well | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | None found on 10 Nov 2026. | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/fabric-native-execution-engine-vs-speedata/ --- # NVIDIA cuDF plugin for Apache Spark and Speedata: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** NVIDIA cuDF plugin for Apache Spark and Speedata are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for NVIDIA cuDF plugin for Apache Spark and 2 entries for Speedata, each with who ran it. ## The same rows for both | Row | NVIDIA cuDF plugin for Apache Spark | Speedata | | --- | --- | --- | | Kind | GPU plugin, formerly RAPIDS Accelerator | PCIe analytics chip plus Dash Spark plugin | | Does well | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. | | Limited | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. | | Published speed results | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. | | Published cost results and prices | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. | | Code changes | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Tuning | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. | | Spark versions, platforms and table formats | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. | | Where it runs | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. | | How open its benchmarks are | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. | | Change Budget: code | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | | Change Budget: cluster | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | | Change Budget: cloud | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | | Change Budget: vendor | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. | | Ledger entries | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) (run by Microsoft (paper); numbers read from Speedata's blog); [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) (run by its maker) | Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded. ## Sources - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) - Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - Speedata: speedata.io/product: https://www.speedata.io/product (read 10 Nov 2026) - Speedata: speedata.io/technology: https://www.speedata.io/technology (read 10 Nov 2026) - Speedata: speedata.io/workload-analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: github.com/Speedata-io: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/nvidia-rapids-spark-vs-speedata/ --- # The Spark Accelerator Ledger: every published benchmark, with who ran it The Ledger, read 10 Nov 2026 **The short answer.** The Spark Accelerator Ledger lists 21 published benchmarks for 12 ways to speed up Spark, newest first. 20 were run by the maker of the option tested and 1 by another named party. Each entry gives the workload, data size, hardware, the result as published and whether the method is public. ## All 21 entries, newest first | Published | Option | Test | Result as published | Ran by | Method published | | --- | --- | --- | --- | --- | --- | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | | 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | | 2026-09 | [Flarion](/options/flarion/) | [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found | | 2026-08-25 | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) | | 2026-06-11 | [DualBird](/options/dualbird/) | [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/) | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found | | 2026-06-11 | [Speedata](/options/speedata/) | [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/) | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched | | 2025-08-21 | [DataPelago](/options/datapelago/) | [Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests](/ledger/datapelago-nucleus-vs-cudf-operators/) | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No | | 2025-05-30 | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No | | 2022 | [Databricks Photon](/options/databricks-photon/) | [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/) | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No | | 2021-11-02 | [Databricks Photon](/options/databricks-photon/) | [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/) | 32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba) | Databricks, audited by the TPC (its maker) | No (a TPC audit report exists under TPC rules; not a repeatable open script) | | Not stated | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) | AWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performance | AWS (its maker) | Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results | | Not stated | [Apache Auron](/options/apache-auron/) | [Auron TPC-DS 1 TB against Apache Spark 3.5.6](/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/) | Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image only | Apache Auron project (its maker) | Yes, tools and full Spark config in the auron-project/tpcds-tools repository | | Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/) | Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queries | Apache Gluten project (its maker) | No | | Not stated | [Apache Gluten with Velox](/options/apache-gluten-velox/) | [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/) | Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queries | Apache Gluten project (its maker) | No | | Not stated | [DualBird](/options/dualbird/) | [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/) | DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not stated | DualBird (its maker) | No | | Not stated | [DualBird](/options/dualbird/) | [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/) | DualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark) | DualBird (its maker) | No | | Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) | Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name | Google Cloud (its maker) | No | | Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) | Google Cloud's wording: 2.5x to 4x speedup over open source Spark | Google Cloud (its maker) | No | | Not stated | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) | NVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as published | NVIDIA (its maker) | No (the NDS automation is public; the comparison is described on the overview page only) | | Not stated | [Speedata](/options/speedata/) | [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/) | Speedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing time | Speedata (its maker) | No | No number here is derived, averaged or estimated. Results from different entries ran on different hardware and data. ## Entries per option | Option | Entries | Run by its maker | Run by another party | | --- | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | 1 | 1 | 0 | | [Apache Auron](/options/apache-auron/) | 1 | 1 | 0 | | [Apache DataFusion Comet](/options/datafusion-comet/) | 2 | 2 | 0 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | 2 | 2 | 0 | | [Databricks Photon](/options/databricks-photon/) | 2 | 2 | 0 | | [DataPelago](/options/datapelago/) | 1 | 1 | 0 | | [DualBird](/options/dualbird/) | 3 | 3 | 0 | | [Flarion](/options/flarion/) | 2 | 2 | 0 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | 3 | 3 | 0 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | 0 | 0 | 0 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | 2 | 2 | 0 | | [Speedata](/options/speedata/) | 2 | 1 | 1 | Alphabetical by option. ## Download and cite CSV: [/datasets/spark-accelerator-ledger.csv](/datasets/spark-accelerator-ledger.csv). JSON: [/datasets/spark-accelerator-ledger.json](/datasets/spark-accelerator-ledger.json). Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0). ## Sources - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Speedata post on the SIGMOD paper: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) - DataPelago Nucleus versus cuDF post: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - Google Lightning Engine launch blog: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks TPC-DS record announcement: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - NVIDIA Spark plugin overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/ --- # How the Ledger is compiled Ledger method, read 10 Nov 2026 **The short answer.** The Spark Accelerator Ledger lists every published benchmark found for the twelve options, with who ran it, what it ran on and what it cost. Each entry carries its source and a publication date. Entries run newest first, and no number is derived, averaged or estimated. ## What counts as a published benchmark An entry is a result a named party has published on its own page, with a link: a dated blog post, a documentation page, a GitHub results page, a paper or a project homepage. A claim without a workload, such as a percentage on a pricing page, is listed as published and noted as having no method stated. Customer quotes that name no baseline and no workload are not benchmarks. Where an option has no benchmark, none found on 2026-11-10 is the entry: Microsoft Fabric's native execution engine has none. ## Dates and order Entries are ordered by publication date, newest first, using the date on the source page. Where only a month or a year is shown, that is what the entry carries. Entries whose pages show no date are marked Not stated and listed last. ## The vendor-run label Each entry says who ran the test. An entry is labelled vendor-run when the party that ran it sells or maintains the option tested, or sells a product in the comparison. Apache projects are named as the project that ran the test. A cloud provider's post written with a vendor is labelled with both names. ## Why no numbers are derived Every figure is quoted as published, in the units the publisher used. The Ledger does not convert tests onto a common scale, compute speedups from published times, or average results across sources, because the tests use different data sizes, hardware, Spark versions and baselines. Two entries can only be compared by reading their hardware and baseline fields. ## Sources - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/method/ --- # Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 Ledger entry, published 2026-11-03 **The short answer.** The Comet project reran its TPC-DS scale factor 1000 test against Spark 4.1.3 on the same EKS cluster. Spark took 1,336 seconds, Comet 1.1.0 took 613 (2.18x) and Comet 1.2.0 took 532 (2.51x). The project says Spark 4.1.3 is slower mainly because of the default S3A input stream. - **Option tested:** [Apache DataFusion Comet](/options/datafusion-comet/) - **Ran by:** Apache DataFusion Comet project (the maker of the option tested) - **Published:** 2026-11-03 - **Workload:** TPC-DS derived queries, 103 queries, Parquet (zstd) in S3 - **Data size:** 1 TB (scale factor 1000) - **Hardware:** Same EKS cluster on r6i instances as the Spark 4.2.0 run - **Result as published:** Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) - **Cost result as published:** None published - **Method published:** Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository ## What the test shows and does not show Spark 4.2.0 with the classic S3A stream took 1,294 s on the same cluster, according to the project. The two baseline rows together show how far a speedup ratio moves with the Spark version. ## Other Ledger entries for Apache DataFusion Comet - [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/): 2026-11-03, run by Apache DataFusion Comet project ## Sources - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/ --- # Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 Ledger entry, published 2026-11-03 **The short answer.** The Apache DataFusion Comet project ran 103 TPC-DS derived queries at scale factor 1000 on an EKS cluster of r6i instances. Spark 4.2.0 took 994 seconds, Comet 1.1.0 took 601 and Comet 1.2.0 took 488. The project publishes scripts and its non-default Spark settings. - **Option tested:** [Apache DataFusion Comet](/options/datafusion-comet/) - **Ran by:** Apache DataFusion Comet project (the maker of the option tested) - **Published:** 2026-11-03 - **Workload:** TPC-DS derived queries, 103 queries, Parquet (zstd) in S3, each query run twice, mean reported - **Data size:** 1 TB (scale factor 1000) - **Hardware:** EKS cluster on r6i instances with EBS; 32 executors x 16 cores, 64G executor memory (Comet adds 32G off-heap), driver 128G - **Result as published:** Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) - **Cost result as published:** None published - **Method published:** Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository ## What the test shows and does not show The derived queries are not comparable to official TPC-DS results, and the node count of the EKS cluster is not stated. The speedup depends on the Spark baseline, as the same page's Spark 4.1.3 run shows. ## Other Ledger entries for Apache DataFusion Comet - [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/): 2026-11-03, run by Apache DataFusion Comet project ## Sources - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/ --- # Flarion TPC-DS SF100, six engines on one machine Ledger entry, published 2026-09 **The short answer.** Flarion's September 2026 benchmarks page compares six engines on TPC-DS (103 queries) at 100 GB on one i4i.8xlarge. Totals: Spark 3.5.5 1,135 s, Comet 652 s, Gluten with Velox 557 s, Flarion 386 s, Databricks Photon 362 s and Databricks Standard 1,180 s. - **Option tested:** [Flarion](/options/flarion/) - **Ran by:** Flarion (the maker of the option tested) - **Published:** 2026-09 - **Workload:** TPC-DS, 103 queries, sum of the lowest per-query times, two runs - **Data size:** 100 GB (SF100) - **Hardware:** One i4i.8xlarge (32 vCPU, 256 GB), us-west-2a, Parquet read from S3; Databricks rows on the same instance type in us-east-1 - **Result as published:** Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s - **Cost result as published:** None published - **Method published:** Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found ## What the test shows and does not show Flarion ran every engine, including its own, and labels the figures as benchmark results, not customer results or an audited TPC result. The page states per-query caps of 600 s at this size. ## Other Ledger entries for Flarion - [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/): 2026-09, run by Flarion ## Sources - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/flarion-tpcds-sf100-six-engines/ --- # Flarion TPC-DS SF1000, six engines on one machine Ledger entry, published 2026-09 **The short answer.** Flarion's September 2026 benchmarks page repeats the six-engine TPC-DS test at 1 TB on one i4i.8xlarge: Spark 3.5.5 135.6 min, Comet 84.4, Gluten with Velox 46.9, Flarion 38.3, Databricks Photon 30.4 and Databricks Standard 103.1. Flarion ran the test itself and caps each query at 1,800 seconds. - **Option tested:** [Flarion](/options/flarion/) - **Ran by:** Flarion (the maker of the option tested) - **Published:** 2026-09 - **Workload:** TPC-DS, 103 queries, sum of per-query times, one run, 1,800 s cap per query - **Data size:** 1 TB (SF1000) - **Hardware:** One i4i.8xlarge (32 vCPU, 256 GB), us-west-2a, Parquet read from S3; Databricks rows on the same instance type in us-east-1 - **Result as published:** Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min - **Cost result as published:** None published - **Method published:** Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found ## What the test shows and does not show Flarion discloses that query q72 is counted at vanilla Spark's time for Gluten and Comet. This size used one run per query, against two runs at 100 GB. ## Other Ledger entries for Flarion - [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/): 2026-09, run by Flarion ## Sources - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/flarion-tpcds-sf1000-six-engines/ --- # GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB Ledger entry, published 2026-08-25 **The short answer.** AWS and NVIDIA published TPC-DS at 3 TB on eight-node EMR on EKS clusters: 281 seconds on g7.4xlarge GPU instances against 1,010 to 1,043 seconds on CPU instances, at $2.06 against $2.93 to $3.18 per run. The post gives the cluster, Spark config and pricing method. - **Option tested:** [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) - **Ran by:** AWS with NVIDIA (the maker of the option tested) - **Published:** 2026-08-25 - **Workload:** TPC-DS, all 103 queries, power run, Parquet on S3, median of 5 iterations - **Data size:** 3 TB - **Hardware:** 8 x g7.4xlarge (1 x NVIDIA RTX PRO 4500 Blackwell, 16 vCPU, 64 GB) against 8 x m9gd.4xlarge, m8id.4xlarge and m8a.4xlarge; EMR on EKS 7.12.0, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1 - **Result as published:** GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x - **Cost result as published:** $2.06 per run on GPU against $2.93 to $3.18 per run on CPU; 'up to 31% less per run than Graviton' - **Method published:** Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) ## What the test shows and does not show Cost uses On-Demand us-east-1 prices as of May 2026 and excludes the EKS control-plane fee, EBS, S3 and the driver pod. It is a cloud-provider post written jointly with the plugin's vendor. ## Other Ledger entries for NVIDIA cuDF plugin for Apache Spark - [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/): Not stated, run by NVIDIA ## Sources - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/ --- # DualBird sort-based Iceberg compaction, 100 GB Ledger entry, published 2026-06-11 **The short answer.** DualBird ran sort-based Iceberg compaction on a synthetic 100 GB table and published about 12x to 20x faster task execution, 55% to 85% lower EC2 cost and about $2 per TB against $6 to over $13 per TB on CPU clusters. The white paper with the method is behind a form. - **Option tested:** [DualBird](/options/dualbird/) - **Ran by:** DualBird (the maker of the option tested) - **Published:** 2026-06-11 - **Workload:** Sort-based Apache Iceberg compaction on a synthetic denormalized event table (about 370 million rows, 64 columns, ZSTD level 3) - **Data size:** 100 GB - **Hardware:** AWS F2 instances for DualBird; baselines are vanilla Spark and an unnamed C++ accelerated Spark; instance types and counts not stated in the public post - **Result as published:** About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) - **Cost result as published:** 55% to 85% lower EC2 cost; about $2 per TB for DualBird against about $6 per TB rising past $13 per TB for CPU-based clusters - **Method published:** Behind a sign-up form (white paper dated 2026-07-01); no public repository found ## What the test shows and does not show DualBird attributes part of the saving to F2 local NVMe replacing EBS volumes. The post also uses the phrases 'higher practical performance ceiling' and 'stable compaction cost across compute scaling ranges'; both are DualBird's wording and neither is a measurement, so they are not in the result cell. The result date (2026-06-11) is the date on the blog post. The white paper that holds the method is dated 2026-07-01, about three weeks later, so the method was released after the result. The home page shows X15 to X20 for the same test (the 'DualBird single task performance tile' entry) while the post says 12x to 20x; instance counts, Spark version and scripts are not public. ## Other Ledger entries for DualBird - [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/): Not stated, run by DualBird - [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/): Not stated, run by DualBird ## Sources - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/dualbird-iceberg-compaction-100gb/ --- # Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper Ledger entry, published 2026-06-11 **The short answer.** Speedata's blog relays Table 2 of a Microsoft SIGMOD 2026 paper on TPC-H at scale factor 100: the Speedata APU was 36.7x on Q07, 40.7x on Q10, 49.4x on Q12, 259.2x on Q13 and 61.5x on Q16, against single-core SQL Server. Microsoft ran the test. - **Option tested:** [Speedata](/options/speedata/) - **Ran by:** Microsoft (paper); numbers read from Speedata's blog (another party) - **Published:** 2026-06-11 - **Workload:** TPC-H queries Q07, Q10, Q12, Q13 and Q16 through Microsoft's coprocessor abstraction layer, standalone APU against single-core SQL Server - **Data size:** TPC-H scale factor 100, Parquet, warm in CPU memory - **Hardware:** Speedata C-200 APU; baseline SQL Server on Intel Xeon Platinum 8473C at 2.80GHz, single core - **Result as published:** APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server - **Cost result as published:** None published - **Method published:** Yes, in the paper; the paper itself was not fetched ## What the test shows and does not show The five queries are a selection, and the baseline is SQL Server on one core, not Spark. The numbers come from Speedata's summary because the paper page could not be fetched. ## Other Ledger entries for Speedata - [Speedata production Spark workload tile](/ledger/speedata-production-spark-consolidation/): Not stated, run by Speedata ## Sources - Speedata post on the SIGMOD paper: https://www.speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-says-about-the-speedata-apu (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/speedata-apu-tpch-sf100-microsoft-paper/ --- # Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests Ledger entry, published 2025-08-21 **The short answer.** DataPelago's August 2025 blog compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge. It reports up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection. - **Option tested:** [DataPelago](/options/datapelago/) - **Ran by:** DataPelago (the maker of the option tested) - **Published:** 2025-08-21 - **Workload:** Project, filter, aggregate, hash aggregate, hash join and K-largest selection operators on a single GPU - **Data size:** Not stated - **Hardware:** AWS p5.48xlarge, AMD EPYC 7R13 (192 vCPU), 2 TB DRAM, 8x NVIDIA H100 (one GPU used) - **Result as published:** Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show These are single-operator tests against a GPU library, not end-to-end Spark jobs. DataPelago's DPA-S plugin documentation does not state a GPU requirement, so the test hardware may differ from what a customer deploys. ## Other Ledger entries for DataPelago None found on 10 Nov 2026. ## Sources - DataPelago Nucleus versus cuDF post: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/datapelago-nucleus-vs-cudf-operators/ --- # Lightning Engine against open source Spark, TPC-H-like 10 TB Ledger entry, published 2025-05-30 **The short answer.** Google Cloud's launch blog of 2025-05-30 reports that Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads at 10 TB, against open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official results. No scripts, cluster spec or raw timings are published. - **Option tested:** [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) - **Ran by:** Google Cloud (the maker of the option tested) - **Published:** 2025-05-30 - **Workload:** TPC-H derived queries (not compliant with the TPC-H specification) - **Data size:** 10 TB - **Hardware:** Open source Spark on 'similar infrastructure'; cluster not published - **Result as published:** 3.6x acceleration over open source Spark - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show The launch was a preview announcement, and the product has since been renamed within Managed Service for Apache Spark. The 3.6x figure does not say which Lightning Engine features were on. ## Other Ledger entries for Google Cloud Lightning Engine - [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/): Not stated, run by Google Cloud - [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/): Not stated, run by Google Cloud ## Sources - Google Lightning Engine launch blog: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/google-lightning-engine-tpch-like-10tb/ --- # Photon customer speedups, SIGMOD 2022 paper Ledger entry, published 2022 **The short answer.** The SIGMOD 2022 Photon paper, written by Databricks engineers, reports that customers saw an average speedup of 3x over the previous Databricks Runtime, with a maximum over 10x. The baseline is Databricks's own earlier Spark-based runtime, not open source Spark, and no customer workloads are named. - **Option tested:** [Databricks Photon](/options/databricks-photon/) - **Ran by:** Databricks authors (the maker of the option tested) - **Published:** 2022 - **Workload:** Customer production queries - **Data size:** Not stated - **Hardware:** Not stated - **Result as published:** Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show The comparison is between two Databricks runtimes, so it does not show a gain over open source Spark. Only the year of publication is recorded here. ## Other Ledger entries for Databricks Photon - [Databricks audited 100 TB TPC-DS result on Photon](/ledger/databricks-photon-tpcds-100tb-audited/): 2021-11-02, run by Databricks, audited by the TPC ## Sources - Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/databricks-photon-customer-speedups-sigmod-2022/ --- # Databricks audited 100 TB TPC-DS result on Photon Ledger entry, published 2021-11-02 **The short answer.** Databricks announced on 2021-11-02 an audited 100 TB TPC-DS result of 32,941,245 QphDS, against 14,861,137 QphDS for Alibaba, the previous record holder. The run used a Databricks SQL warehouse, which the post says is based on Photon rather than Apache Spark. - **Option tested:** [Databricks Photon](/options/databricks-photon/) - **Ran by:** Databricks, audited by the TPC (the maker of the option tested) - **Published:** 2021-11-02 - **Workload:** Official TPC-DS run on Databricks SQL - **Data size:** 100 TB - **Hardware:** Databricks SQL on AWS with Delta Lake on S3 (per the Photon paper) - **Result as published:** 32,941,245 QphDS at 100 TB, against 14,861,137 QphDS for the previous record holder (Alibaba) - **Cost result as published:** None published - **Method published:** No (a TPC audit report exists under TPC rules; not a repeatable open script) ## What the test shows and does not show The result is for a SQL warehouse, not a Spark batch job, and it cannot be repeated with an open script. The TPC audit covers the run's compliance, not how Photon performs on a team's own Spark jobs. ## Other Ledger entries for Databricks Photon - [Photon customer speedups, SIGMOD 2022 paper](/ledger/databricks-photon-customer-speedups-sigmod-2022/): 2022, run by Databricks authors ## Sources - Databricks TPC-DS record announcement: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/databricks-photon-tpcds-100tb-audited/ --- # EMR runtime claim on the AWS features page Ledger entry, published Not stated **The short answer.** AWS's Amazon EMR features page says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents and delivers 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale, the cluster or the workload behind either figure. - **Option tested:** [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) - **Ran by:** AWS (the maker of the option tested) - **Published:** Not stated - **Workload:** Not stated on the page (Iceberg write also cited) - **Data size:** Not stated - **Hardware:** Not stated - **Result as published:** AWS's wording: up to 4.5x faster than open-source equivalents; 2.7x faster Iceberg write performance - **Cost result as published:** None published - **Method published:** Partly: AWS publishes the aws-samples/emr-spark-benchmark harness, which states no results ## What the test shows and does not show Undated; AWS publishes a benchmark harness but the 4.5x and 2.7x figures name no scale or cluster. AWS publishes a 3 TB TPC-DS harness on GitHub, but the repository lists commands and configs rather than results. A per-version result post with cost for the same work: none found on 2026-11-10. ## Other Ledger entries for Amazon EMR runtime for Apache Spark None found on 10 Nov 2026. ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/amazon-emr-runtime-features-page-claim/ --- # Auron TPC-DS 1 TB against Apache Spark 3.5.6 Ledger entry, published Not stated **The short answer.** The Apache Auron project ran TPC-DS at 1 TB on Spark 3.5.6 against Auron 6.0.0-preview on YARN. It publishes a claim of about 2x faster and about 50% fewer cluster resources, with per-query results only as a chart image. Node types and counts are not given. - **Option tested:** [Apache Auron](/options/apache-auron/) - **Ran by:** Apache Auron project (the maker of the option tested) - **Published:** Not stated - **Workload:** TPC-DS, Parquet with zstd, run with a spark-sql-perf derived runner - **Data size:** 1 TB (scale factor 1000) - **Hardware:** YARN cluster, executor cores 8; node type and count not published - **Result as published:** Published as 'running ~2x faster and save ~50% cluster resources' against Apache Spark 3.5; per-query results are a chart image only - **Cost result as published:** None published (only the relative claim of about 50% fewer cluster resources) - **Method published:** Yes, tools and full Spark config in the auron-project/tpcds-tools repository ## What the test shows and does not show The page publishes the data generator, runner and full Spark config, so the method can be repeated. Per-query numbers are not in text form, and the benchmark used Auron 6.0.0-preview, not the current 8.0.0-incubating release. ## Other Ledger entries for Apache Auron None found on 10 Nov 2026. ## Sources - Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/apache-auron-tpcds-1tb-vs-spark-3-5/ --- # Gluten with Velox, TPC-DS 3 TB (project homepage) Ledger entry, published Not stated **The short answer.** The Apache Gluten project homepage reports TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: 3.02x overall and up to 13.75x on single queries. As with its TPC-H tile, the cluster size is not stated and no method is linked. - **Option tested:** [Apache Gluten with Velox](/options/apache-gluten-velox/) - **Ran by:** Apache Gluten project (the maker of the option tested) - **Published:** Not stated - **Workload:** TPC-DS - **Data size:** 3 TB - **Hardware:** Intel Xeon Platinum 8592+, Spark 3.3.1 (cluster size not stated) - **Result as published:** Gluten project's wording: overall 3.02x speedup with Gluten and Velox; up to 13.75x on individual queries - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. The gluten-it tool in the repository can run TPC-style queries, but the homepage numbers do not link a run configuration. The same page reports a separate TPC-H result at the same data size. ## Other Ledger entries for Apache Gluten with Velox - [Gluten with Velox, TPC-H 3 TB (project homepage)](/ledger/apache-gluten-velox-tpch-3tb/): Not stated, run by Apache Gluten project ## Sources - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/apache-gluten-velox-tpcds-3tb/ --- # Gluten with Velox, TPC-H 3 TB (project homepage) Ledger entry, published Not stated **The short answer.** The Apache Gluten project homepage reports TPC-H at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: 3.34x overall and up to 23.45x on single queries. The cluster size is not stated, and no method or scripts are linked from the page. - **Option tested:** [Apache Gluten with Velox](/options/apache-gluten-velox/) - **Ran by:** Apache Gluten project (the maker of the option tested) - **Published:** Not stated - **Workload:** TPC-H - **Data size:** 3 TB - **Hardware:** Intel Xeon Platinum 8592+, Spark 3.3.1 (cluster size not stated) - **Result as published:** Gluten project's wording: overall 3.34x speedup with Gluten and Velox; up to 23.45x on individual queries - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. The hardware is from Intel, a Gluten contributor, and Spark 3.3.1 is older than the versions the docs now list as officially supported. The page shows no baseline configuration, so the ratio cannot be reproduced from it. ## Other Ledger entries for Apache Gluten with Velox - [Gluten with Velox, TPC-DS 3 TB (project homepage)](/ledger/apache-gluten-velox-tpcds-3tb/): Not stated, run by Apache Gluten project ## Sources - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/apache-gluten-velox-tpch-3tb/ --- # DualBird 110 GB enrichment pipeline (join denormalization) Ledger entry, published Not stated **The short answer.** DualBird's home and product pages show a 110 GB enrichment pipeline with join denormalization: a 6x to 18x higher performance ceiling and 50% to 85% lower EC2 cost. The pages give no hardware, baseline, date or method, and no write-up was found. - **Option tested:** [DualBird](/options/dualbird/) - **Ran by:** DualBird (the maker of the option tested) - **Published:** Not stated - **Workload:** Enrichment pipeline with join denormalization - **Data size:** 110 GB - **Hardware:** Not stated - **Result as published:** DualBird's wording: '6x to 18x higher performance ceiling' on a 110 GB enrichment pipeline; baseline, hardware and absolute times not stated - **Cost result as published:** 50% to 85% lower EC2 cost - **Method published:** No ## What the test shows and does not show Home-page marketing tile: no test described and no date. It is kept because it is DualBird's only published result for a join-heavy workload. With no baseline stated, the ratio cannot be placed against any other entry. ## Other Ledger entries for DualBird - [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/): 2026-06-11, run by DualBird - [DualBird single task performance tile](/ledger/dualbird-single-task-speedup/): Not stated, run by DualBird ## Sources - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/dualbird-enrichment-pipeline-110gb/ --- # DualBird single task performance tile Ledger entry, published Not stated **The short answer.** DualBird's home page shows a Single Task Performance tile reading X15-X20 speedup for the same cluster size, with cost reduced by over 50%. The tile appears to restate the compaction test, whose post says 12x to 20x, and the baseline is described differently on different DualBird pages. - **Option tested:** [DualBird](/options/dualbird/) - **Ran by:** DualBird (the maker of the option tested) - **Published:** Not stated - **Workload:** Single Spark task at the same cluster size (home page tile; appears to be the compaction workload) - **Data size:** Not stated - **Hardware:** Not stated - **Result as published:** DualBird's wording: 'X15-X20 speedup for the same cluster size'; baseline not stated on the tile (the compaction post names vanilla Spark and an unnamed C++ accelerated Spark) - **Cost result as published:** Reducing cost by over 50% - **Method published:** No ## What the test shows and does not show Home-page marketing tile: no test described and no date. It appears to restate the 100 GB compaction entry above, which is the entry with the source post. The baseline is vanilla Spark in the compaction post and a high-end C++ columnar engine on a single CPU core in an April 2026 post. ## Other Ledger entries for DualBird - [DualBird sort-based Iceberg compaction, 100 GB](/ledger/dualbird-iceberg-compaction-100gb/): 2026-06-11, run by DualBird - [DualBird 110 GB enrichment pipeline (join denormalization)](/ledger/dualbird-enrichment-pipeline-110gb/): Not stated, run by DualBird ## Sources - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/dualbird-single-task-speedup/ --- # Lightning Engine claim on Google's pricing page Ledger entry, published Not stated **The short answer.** Google Cloud's pricing page for Managed Service for Apache Spark says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance of an unnamed high-speed Spark alternative. No workload, method or competitor is named on the page. - **Option tested:** [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) - **Ran by:** Google Cloud (the maker of the option tested) - **Published:** Not stated - **Workload:** Not stated - **Data size:** Not stated - **Hardware:** Not stated - **Result as published:** Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name - **Cost result as published:** None published (the claim is relative price-performance) - **Method published:** No ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. The page carries no date, so the publication date is not stated. The same page lists the DCU rates that a reader can use to price their own jobs. ## Other Ledger entries for Google Cloud Lightning Engine - [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/): 2025-05-30, run by Google Cloud - [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/): Not stated, run by Google Cloud ## Sources - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/google-lightning-engine-pricing-page-claim/ --- # Native Query Execution on Axion ARM, Google's documentation Ledger entry, published Not stated **The short answer.** Google Cloud's Lightning Engine documentation claims that Native Query Execution on Google Axion ARM (C4A) machines delivers a 2.5x to 4x speedup over open source Spark on analytical workloads. The page gives no workload, data size, cluster or method, and names no one who ran the test. - **Option tested:** [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) - **Ran by:** Google Cloud (the maker of the option tested) - **Published:** Not stated - **Workload:** Analytical workloads (not specified) - **Data size:** Not stated - **Hardware:** Google Axion ARM (C4A) machines - **Result as published:** Google Cloud's wording: 2.5x to 4x speedup over open source Spark - **Cost result as published:** None published - **Method published:** No ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. The claim is in product documentation, not in a dated benchmark post. The same page lists jobs for which NQE is not recommended, including RDD-heavy jobs and UDFs. ## Other Ledger entries for Google Cloud Lightning Engine - [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/): 2025-05-30, run by Google Cloud - [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/): Not stated, run by Google Cloud ## Sources - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/google-native-query-execution-axion/ --- # NVIDIA comparison with open-source Spark on EC2, NDS 3 TB Ledger entry, published Not stated **The short answer.** NVIDIA's overview page compares the AWS GPU result with its own 3 TB NDS runs of open-source Spark 3.5.6 on EC2. It publishes the GPU run as 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. - **Option tested:** [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) - **Ran by:** NVIDIA (the maker of the option tested) - **Published:** Not stated - **Workload:** NDS (TPC-DS derived), 103 queries - **Data size:** 3 TB - **Hardware:** GPU result from the AWS EMR run; CPU runs on EC2 m7gd.8xlarge and m9gd.4xlarge with open-source Spark 3.5.6 - **Result as published:** NVIDIA's wording: GPU run 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, as published - **Cost result as published:** 77% and 72% lower reported cost per run; GPU cost includes EC2 and EMR on EKS, CPU costs include EC2 only - **Method published:** No (the NDS automation is public; the comparison is described on the overview page only) ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. NVIDIA itself notes that the runs use different hardware and software stacks. The GPU cost includes EC2 and EMR on EKS while the CPU costs include EC2 only. ## Other Ledger entries for NVIDIA cuDF plugin for Apache Spark - [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/): 2026-08-25, run by AWS with NVIDIA ## Sources - NVIDIA Spark plugin overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/ --- # Speedata production Spark workload tile Ledger entry, published Not stated **The short answer.** Speedata's home page shows customer-result tiles: 62.7x acceleration on production Spark workloads, 37 servers consolidated to 3, and processing time from 90 hours to 8 on the same hardware budget. The customer is not named, and no data size, hardware or method is given. - **Option tested:** [Speedata](/options/speedata/) - **Ran by:** Speedata (the maker of the option tested) - **Published:** Not stated - **Workload:** Enterprise AI data preprocessing on Spark; customer not named - **Data size:** Not stated - **Hardware:** Not stated - **Result as published:** Speedata's wording: 62.7x acceleration on production Spark workloads; 37 servers consolidated to 3; 90 to 8 hours of processing time - **Cost result as published:** None published (the tile claims up to 90% lower TCO) - **Method published:** No ## What the test shows and does not show Undated and no method published: the page shows no date and no workload description. The tiles describe enterprise AI data preprocessing, and the home page also cites 52x more performance per dollar than a GPU on an unnamed industry-standard benchmark. No date is shown, so the publication date is not stated. ## Other Ledger entries for Speedata - [Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper](/ledger/speedata-apu-tpch-sf100-microsoft-paper/): 2026-06-11, run by Microsoft (paper); numbers read from Speedata's blog ## Sources - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/ledger/speedata-production-spark-consolidation/ --- # The Change Budget: what each option makes a team change Change Budget, read 10 Nov 2026 **The short answer.** The Change Budget sets out, for each of the 12 options, what a team has to change in four places: job code, the cluster (instances, plugins, configuration), the cloud it must run on, and the vendor or platform it must contract with. Every cell carries its source and read date, 10 Nov 2026. ## Four columns, twelve options | Option | Code | Cluster | Cloud | Vendor | | --- | --- | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | **No code change stated.** AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | **Run the jobs on an EMR release.** Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | **AWS only.** AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | **EMR charges added to EC2 and EBS.** The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | | [Apache Auron](/options/apache-auron/) | **No job code change stated.** Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | **Build the jar, set three configs.** You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | **None found on 2026-11-10.** No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | **Apache-2.0, no contract stated.** Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | | [Apache DataFusion Comet](/options/datafusion-comet/) | **No job code change stated.** Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | **Plugin, shuffle manager, off-heap memory.** Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | **None found on 2026-11-10.** No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | **Apache-2.0, no contract stated.** Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | **No DataFrame or SQL changes.** Gluten needs no DataFrame or SQL changes, only configuration. | **Plugin, off-heap memory, shuffle manager.** The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | **None found on 2026-11-10.** No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | **Apache-2.0, no contract stated.** Gluten's project license is Apache-2.0; no vendor or platform contract is described. | | [Databricks Photon](/options/databricks-photon/) | **No change, with listed exceptions.** Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | **Set runtime_engine to PHOTON in API.** Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | **Databricks on AWS, Azure or GCP.** Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | **Databricks platform required.** Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. | | [DataPelago](/options/datapelago/) | **No application rewrite stated.** DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | **Plugin, license and memory settings.** Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | **AWS EKS or Google Cloud.** DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | **Marketplace subscription required.** DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. | | [DualBird](/options/dualbird/) | **No code change stated.** DualBird's about page says 'No infrastructure changes. No code changes.' | **Instance type change to F2.** DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | **AWS only, F2 in eight regions.** F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | **Plugin contract; no price published.** No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. | | [Flarion](/options/flarion/) | **No code change stated.** Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | **One JAR and one config change.** Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | **No single cloud required.** Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | **Flarion license at usage-based prices.** Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | **No Spark code change stated.** Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | **Image 2.3.3+ or 3.0.2+, engine flag.** Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | **Google Cloud only.** Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | **Premium tier or cluster add-on.** Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | **No code modifications stated.** Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | **Toggle in environment or Spark config.** Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | **Microsoft Fabric on Azure only.** The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | **Fabric capacity required.** Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | **No code change stated.** NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | **GPU instances and plugin settings.** The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | **AWS, Azure and Google Cloud documented.** NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | **Free software; GPU instances paid.** The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. | | [Speedata](/options/speedata/) | **No code change stated.** Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | **PCIe C200 card in each server.** The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | **On-premises, OEM or hosted AWS.** Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | **Speedata purchase; no price list.** No price list is published. The pages offer buying a server or going through OEM or cloud partners. | Alphabetical by option. Sources for each cell are on the option's profile and below. ## Download and cite CSV: [/datasets/change-budget.csv](/datasets/change-budget.csv). JSON: [/datasets/change-budget.json](/datasets/change-budget.json). Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0). ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR Spark performance guide: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Apache Auron homepage: https://auron.apache.org/ (read 10 Nov 2026) - Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Auron README: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Comet compatibility guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Comet tuning guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Comet installation guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) - Apache Gluten getting started: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Gluten downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - Apache Gluten Powered By page: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Databricks pricing page: https://www.databricks.com/product/pricing (read 10 Nov 2026) - DataPelago Accelerator for Spark page: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DataPelago AWS deployment docs: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DataPelago Google Cloud deployment docs: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DualBird about page: https://dualbird.io/about (read 10 Nov 2026) - DualBird product page: https://dualbird.io/product (read 10 Nov 2026) - AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - Flarion pricing page: https://www.flarion.io/pricing (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Microsoft Fabric pricing: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - NVIDIA Spark plugin overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/change-budget/ --- # How the Change Budget is built Change Budget method, read 10 Nov 2026 **The short answer.** The Change Budget lists what each of the twelve options asks a team to change: job code, cluster, cloud and vendor. Each cell has a plain label, one or two sentences and a source read on 2026-11-10. Where nothing was found, the label says None found on 2026-11-10. No cell is given a number or added up. ## What the four columns mean - Code: changes to job code, such as rewriting queries or replacing UDFs. A vendor statement of no code change is recorded with its source, along with any listed exceptions. - Cluster: changes to the cluster, instances, configuration or plugins, such as a jar, off-heap memory settings or a different instance type. - Cloud: which clouds the option requires, from the vendor's own documentation or the cloud provider's. - Vendor: whether a new vendor or platform contract is required, for example a Databricks workspace, a marketplace subscription or a hardware purchase. ## How each cell is sourced Every cell cites the vendor's documentation, an Apache project page or a cloud provider page, and carries the date it was read. Statements such as no code change are the vendor's own and are written as such. They are not tested here. Where a page gave no answer, the label reads None found on 2026-11-10 and the text says what was checked. A missing answer is not a statement that the option cannot do it. ## What the table does not do It does not add the columns into a total, order the rows by effort or say which change is easier. A one-line config change and a hardware purchase are both listed as published, and the reader weighs them against the cluster in front of them. Rows are alphabetical by option name. ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - DualBird product page: https://dualbird.io/product (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/change-budget/method/ --- # How this site works Method, read 10 Nov 2026 **The short answer.** Spark Acceleration Compare lists the twelve ways to speed up Spark that it covers and does not rank them. Every fact has a source link and a read date, and is labelled as a vendor claim or a third-party statement. When nothing is found it says none found on 2026-11-10. Options are alphabetical and the Ledger runs by publication date. ## What the site does The site lists facts about each option side by side: what it is, where it runs, what it changes, what it costs and what has been published about its speed. It does not give marks, set an order of merit or recommend. Where a reader has to choose, the situation pages say what to check and which published facts answer it. ## How facts are sourced, dated and labelled Facts come from the option's own documentation and pages, Apache project pages and repositories, cloud provider documentation and papers. Each fact carries the URL and the date it was read. A claim made by the seller is written as theirs, for example 'DualBird says' or 'per Databricks's documentation'. Statements by a cloud provider or an Apache project are named as such. A statement is labelled third-party only when the party that made it neither sells nor maintains the option tested. Where a test was run by one party and relayed by another, both are named. ## The none-found rule When a page was searched and the answer was not there, the entry reads none found on 2026-11-10 and names what was searched. That wording records a search on one day. It does not say the option cannot do the thing. ## The benchmark rule A benchmark is listed only if a named party published the numbers on a page that can be linked. The entry quotes the numbers as published, names who ran the test, and gives the date. No number is calculated, averaged across sources or estimated. The [Ledger method](/ledger/method/) page has the full rule. ## Order - Options are listed alphabetically by name wherever they appear. - The Ledger is ordered by publication date, newest first, with undated entries last. - Comparison pages place the two options in alphabetical order. ## Corrections A reader can report a wrong or outdated fact to editor@sparkaccelerationcompare.com. The [about page](/about/) explains how corrections are dated and shown. ## Sources - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - DualBird product page: https://dualbird.io/product (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/method/ --- # About Spark Acceleration Compare About, read 10 Nov 2026 **The short answer.** Spark Acceleration Compare is a reference publication on ways to make Apache Spark faster without rewriting jobs. It covers twelve options, from Apache plugins to GPU, FPGA and custom-silicon accelerators and cloud runtimes, and keeps two original tables: the Spark Accelerator Ledger of published benchmarks and the Change Budget of what each option makes a team change. ## What it publishes Option pages with sourced facts, side-by-side comparisons of two options, five situation pages, guides, a glossary, news items with dates, an FAQ and downloadable datasets. The [Ledger](/ledger/) and the [Change Budget](/change-budget/) are the two original tables. ## How it is made Each fact is taken from a page that was fetched and read on a stated date, and the page is linked. The [method page](/method/) sets out the sourcing, labelling and ordering rules. The site lists options and does not rank them. ## Who it is for The reader it is written for is a data engineer who owns a Spark cluster and its bill, and wants to see what is published about each option before running a trial. ## Sources - Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Apache Auron: auron.apache.org: https://auron.apache.org/ (read 10 Nov 2026) - Apache DataFusion Comet: datafusion.apache.org/comet: https://datafusion.apache.org/comet/ (read 10 Nov 2026) - Apache Gluten with Velox: gluten.apache.org: https://gluten.apache.org/ (read 10 Nov 2026) - Databricks Photon: databricks.com: https://www.databricks.com/ (read 10 Nov 2026) - DataPelago: datapelago.ai: https://www.datapelago.ai/ (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026) - Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Microsoft Fabric native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - Speedata: speedata.io: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/about/ --- # Editorial policy Editorial policy, read 10 Nov 2026 **The short answer.** Every fact on this site comes from a page that was fetched and read on a stated date, and each fact links to it. Vendor claims are labelled as the vendor's. The site lists options alphabetically and does not rank them. Corrections are logged with their date, and the none-found rule applies where a search finds nothing. ## What is included An option is included if it speeds up existing Apache Spark jobs without requiring the job code to be rewritten, and has a public page describing it. The twelve options are listed on the [options page](/options/). ## How claims are written - A claim by the seller is attributed to the seller. - A number appears only if a named party published it, with the link and date. - No number is derived, averaged or estimated. - Language that orders options by merit or tells a reader what to choose is not used, in titles or in body text. ## Updates Pages carry the date their facts were read. When a source changes or a correction is accepted, the fact is revised and the correction is logged with its date. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/editorial-policy/ --- # Corrections Corrections, read 10 Nov 2026 **The short answer.** To report a wrong or outdated fact, write to editor@sparkaccelerationcompare.com with the page, the sentence and a link to the source that shows the correct fact. Accepted corrections are made on the page, and the correction is logged with the date it was made and the fact that changed. ## How to report Email editor@sparkaccelerationcompare.com. Include the page address, the text that is wrong, and the source page that shows the right answer. A vendor, a project maintainer or a reader may write. ## What happens next The claim is checked against the source page on the day it is read. If the page supports the correction, the fact is changed and the page shows the correction with its date. If it does not, the reply names the page that was read. ## How corrections are dated Each correction carries the date it was made, in the form 2026-11-10, and a one-line note of what changed. Earlier wording is described in the note and not deleted from the log. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/corrections/ --- # Contact Contact, read 10 Nov 2026 **The short answer.** Write to editor@sparkaccelerationcompare.com for corrections, questions about a source, or a request for a missing option. Include the page address and, for a correction, a link to the page that shows the right fact. Replies name the page that was read and the date it was read. ## Email editor@sparkaccelerationcompare.com. This is the only address the site publishes. ## What to include - The page address. - The sentence in question. - A link to the source page that shows the right answer. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/contact/ --- # Privacy Privacy, read 21 Nov 2026 **The short answer.** Spark Acceleration Compare uses Google Analytics 4, which sets cookies, to count visits. The host also keeps standard access logs through CloudFront, used only to count page views and crawler fetches. The site has no accounts, no forms that collect personal data and no advertising trackers or tracking pixels. Email to the editor is kept to answer it. ## Google Analytics Spark Acceleration Compare uses Google Analytics 4 to count visits. Through cookies, it records the pages viewed, the page that sent the visitor (the referrer), the device and browser, and an approximate location worked out from the IP address. Google processes this data; Google's page [How Google uses information from sites or apps that use our services](https://policies.google.com/technologies/partner-sites) describes how. To opt out, install the [Google Analytics opt-out browser add-on](https://tools.google.com/dlpage/gaoptout) (https://tools.google.com/dlpage/gaoptout), available for Chrome, Safari, Firefox and Microsoft Edge, or block cookies for this site in the browser's settings. ## Access logs CloudFront access logs record the request time, the page requested, the response code and the visitor's IP address and browser string. They are used only to count page views and crawler fetches. ## Email If you write to editor@sparkaccelerationcompare.com, the message and your address are kept to reply and to log any correction. ## Sources - Google, How Google uses information from sites or apps that use our services: https://policies.google.com/technologies/partner-sites (read 21 Nov 2026) - Google Analytics opt-out browser add-on: https://tools.google.com/dlpage/gaoptout (read 21 Nov 2026) Last reviewed 21 Nov 2026. Canonical: https://sparkaccelerationcompare.com/privacy/ --- # Terms of use Terms, read 10 Nov 2026 **The short answer.** The pages on Spark Acceleration Compare are provided for reference. Facts are read from the sources linked beside them on the date shown and may have changed since. Product names belong to their owners. Check the linked source before relying on a fact for a purchase, a migration or a contract. ## Use of the content You may read, link to and quote short passages of the site with a link back. Datasets on the [datasets page](/datasets/) are offered for reuse with the source links kept. ## Accuracy Each fact shows its source and the date it was read. Sources change, so a fact can be out of date. Report problems through the [about page](/about/). ## Links Links to other sites are given as sources. The site does not control those pages. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/terms/ --- # Guides to speeding up Spark without rewriting jobs Guides, read 10 Nov 2026 **The short answer.** Guides explain the parts of Spark acceleration a buyer meets: how native engines and plugins work, what fallback means, how to read a TPC-DS result, what a GPU or FPGA changes in a cluster, and how to test one job. Every guide cites sources with read dates. ## All 10 guides, A to Z - [GPU vs FPGA for Spark: the published facts on each](/guides/gpu-vs-fpga-for-spark/): GPU versus FPGA for Apache Spark: what DualBird and NVIDIA have each published about speed, cost, hardware, operators and availability, side by side with sources. - [GPU, FPGA and custom silicon for Spark in plain words](/guides/hardware-acceleration-plain-words/): What FPGA, GPU and custom-silicon accelerators do for Apache Spark, with DualBird on AWS F2, NVIDIA cuDF and Speedata C200 as examples, plus AWS facts. - [How to read a vendor Spark benchmark: seven checks](/guides/reading-a-vendor-benchmark/): Seven checks for a Spark acceleration benchmark: who ran it, workload, scale factor, hardware, baseline Spark version, cost basis and published method. - [How to run a fair two-week Spark accelerator trial](/guides/trialing-an-accelerator/): A two-week plan to trial a Spark accelerator on your own jobs: baseline, same data, same cluster cost, cost per job, event logs and the Spark UI, with sources. - [Native Spark engines: Auron, DataFusion, Photon, Velox](/guides/native-engines-explained/): What vectorized native Spark engines do (Auron, DataFusion, Photon, Velox): columnar execution, operator coverage, fallback, Spark versions and where each runs. - [Operator coverage and fallback: checking what an accelerator covers](/guides/operator-coverage-and-fallback/): How to check which Spark operators an accelerator covers and what happens on fallback, using each project's supported-operator lists, explain output and the Spark UI. - [Spark acceleration options on Amazon EMR and EKS](/guides/spark-acceleration-on-emr-and-eks/): Which Spark accelerators run on Amazon EMR and EKS and what each needs: instance families, plugins, EMR release labels, Spark versions and billing. - [Spark performance tuning: what to do when it is not enough](/guides/when-tuning-is-cheaper/): Spark performance tuning checklist (AQE, shuffle partitions, broadcast joins, file sizes, skew) and how to tell when tuning is not enough. - [What "no code change" means for Spark acceleration](/guides/what-no-code-change-means/): What no code change means for Spark acceleration: SQL and DataFrame versus RDD and UDF code, fallback, config flags and instance changes, per option. - [Where a Spark job spends its time: scan to JVM](/guides/where-spark-time-goes/): Where Spark time goes: scan, shuffle, sort, join, spill, serialization and the JVM, with figures from the Berkeley NSDI 2015 study and the Apache Spark documentation. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/ --- # Where a Spark job spends its time Guide, read 10 Nov 2026 **The short answer.** A Spark job spends time in scan, shuffle, sort, join, spill, serialization and JVM garbage collection. Published figures disagree on which dominates: a 2015 Berkeley study found network improvements cut job time by a median of at most 2%, while the Spark tuning guide calls network the usual limit. ## Why the split matters before you buy anything Every acceleration option on this site attacks one or more of the same stages. A native engine replaces the JVM operators. A GPU or FPGA takes over sort, join and aggregation. A tuning change shrinks shuffle or skips files. If you do not know where your own jobs spend time, you cannot tell which published speedup applies to you. This guide lists the stages, what the Apache Spark documentation says about each, and the few published measurements of where time goes. It gives no pick and no estimate for your cluster. The last section says what to read in the Spark UI to get your own split. ## The stages of a typical SQL job A Spark SQL query is planned by Catalyst, cut into stages at each shuffle, and run as tasks on executor JVMs. Time in a stage divides roughly into the parts below. Each part has a metric in the Spark UI. - **Scan and decode.** Reading Parquet or ORC from S3 or HDFS, decompressing it and turning columns into rows. Spark packs files into read partitions of up to 128 MB (spark.sql.files.maxPartitionBytes), and charges 4 MB of estimated cost to open each file (spark.sql.files.openCostInBytes), which is why many small files are slow. - **Shuffle write and read.** Data is partitioned, written to local disk by map tasks and fetched over the network by reduce tasks. The default is 200 shuffle partitions (spark.sql.shuffle.partitions). The UI shows shuffle read and write size and the time spent writing shuffle data. - **Sort and join.** Sort-merge joins sort both sides; hash joins build a table in memory. Broadcast joins skip the shuffle for tables under 10 MB by default (spark.sql.autoBroadcastJoinThreshold). - **Aggregation and expression evaluation.** Group-bys, window functions, casts, string and date functions, run row by row or by generated code on the JVM. - **Spill.** When a sort, join or aggregation does not fit its memory, Spark writes it to disk. The UI reports "shuffle spill (memory)" as the deserialized size and "shuffle spill (disk)" as the serialized size. - **Serialization.** Task closures, shuffle records and cached data are serialized. Spark defaults to Java serialization, which its tuning guide calls "often quite slow". Kryo is described there as "significantly faster and more compact than Java serialization (often as much as 10x)", but it needs class registration and is not the default. - **JVM garbage collection.** The task table in the UI carries a GC time column, the total JVM garbage collection time for the task. Spark sets aside 60% of the heap after a 300 MiB reserve for execution and storage by default (spark.memory.fraction 0.6); the rest holds user data structures and metadata. ## What published measurements say about the split The most cited measurement is the NSDI 2015 paper "Making Sense of Performance in Data Analytics Frameworks" from UC Berkeley, which instrumented Spark on two SQL benchmarks and one production workload. Its stated findings: CPU, not I/O, is often the bottleneck; improving network performance can improve job completion time by a median of at most 2%; and the causes of most stragglers can be identified. The paper dates from 2015, so it measured a much older Spark than the current release. The Spark tuning guide, read 2026-11-10, says something different in its summary: because of the in-memory nature of most Spark computations, programs can be bottlenecked by CPU, network bandwidth or memory, and "most often, if the data fits in memory, the bottleneck is network bandwidth". Both statements are published; neither is a measurement of your job. A shuffle-heavy job on a small cluster with slow networking can look like the guide. A cached, compute-heavy job can look like the paper. The Photon paper (SIGMOD 2022, Databricks) takes the CPU-bound reading: it describes replacing the JVM execution layer with a C++ vectorized engine, and reports customer queries averaging 3x faster than the previous Databricks Runtime, with a maximum above 10x. Databricks documents that queries shorter than about two seconds see no meaningful gain, which is a reminder that fixed costs such as planning and scheduling do not shrink with a faster operator. ## Which option attacks which stage | Stage | What changes it, per the project's own documentation | | --- | --- | | Scan and decode | Native Parquet readers in Apache DataFusion Comet and Apache Gluten with Velox; Databricks Photon; NVIDIA's cuDF plugin reads on the GPU | | Shuffle | Columnar shuffle managers (Apache DataFusion Comet, Apache Gluten); remote shuffle services Celeborn and Uniffle (Apache Auron, Apache Gluten); NVIDIA's RAPIDS Shuffle Manager | | Sort, join, aggregation | FPGA (DualBird says its F2 path covers sort, join and aggregation), GPU kernels (NVIDIA), Speedata's APU, native C++ or Rust operators | | Spill and memory | Native engines use off-heap memory that the JVM does not manage, so they need off-heap settings; DualBird says disk spills effectively disappear on its path (DualBird's claim, not independently tested) | | Serialization and GC | Native engines keep data in columnar buffers outside the heap; Flarion says part of a job, including the driver and often the final write, still runs on the JVM heap | Mapping from stage to the mechanism each project documents. Sources are listed at the end of the guide. ## How to get your own split from the Spark UI Turn on event logs before you run: spark.eventLog.enabled defaults to false. With the logs, the History Server rebuilds the same UI after the application ends. Then read three places. On the Stages tab, sort stages by duration and look at the three longest. For each, compare shuffle read size, shuffle write size and the two spill figures against the input size. A stage that spills many times its input is memory-bound. On the task table, compare GC time with task duration, and scheduler delay and result serialization time with the median task. On the SQL tab, find the join type Spark chose and whether AQE changed it. Record the split for your five most expensive jobs before any trial. The guide on [running a two-week trial](/guides/trialing-an-accelerator/) uses the same numbers as its baseline, and the guide on [tuning](/guides/when-tuning-is-cheaper/) shows which stage each setting touches. ## Sources - Making Sense of Performance in Data Analytics Frameworks, NSDI 2015: https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/ousterhout (read 10 Nov 2026) - Apache Spark documentation: Tuning Spark: https://spark.apache.org/docs/latest/tuning.html (read 10 Nov 2026) - Apache Spark documentation: SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026) - Apache Spark documentation: Web UI: https://spark.apache.org/docs/latest/web-ui.html (read 10 Nov 2026) - Photon: A Fast Query Engine for Lakehouse Systems, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Databricks documentation: What is Photon: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Flarion FAQs: https://www.flarion.io/faqs (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/where-spark-time-goes/ --- # What "no code change" has to mean Guide, read 10 Nov 2026 **The short answer.** Every option on this site says no code change, but each means something different: SQL and DataFrame jobs only, fallback to the JVM for UDFs and RDDs, a new jar and config flags, or a new instance type. Photon documents no UDF, RDD or Dataset support; Google requires image 2.3.3 or later. ## Four things the phrase can cover Vendors and projects use "no code change" for four different promises. Read the documentation for which one is being made before you compare two options on it. - **Your job source is untouched.** The same PySpark, Scala or SQL runs. This is true for every option below. - **Your job runs accelerated only if it uses certain APIs.** Spark SQL and the DataFrame API pass through Catalyst, which the accelerators hook into. RDD code, Dataset lambdas and most UDFs do not, so those stages run on the JVM unchanged and unaccelerated. - **Your configuration changes.** A plugin class, a shuffle manager, off-heap memory, executor overhead and a jar on the classpath. Your code is the same, your submit configuration is not. - **Your infrastructure changes.** A different instance family, a different runtime image, a different cloud service or a physical card. The job is the same and the bill is different. ## SQL and DataFrame code versus RDDs and UDFs Spark plans SQL and DataFrame operations as a tree of operators that a plugin can replace one by one. A Scala or Python function passed to map, flatMap, foreachPartition or a UDF is opaque to the planner. Databricks documents that Photon does not support UDFs, RDD APIs or Dataset APIs. DataPelago's documentation tells users to avoid heavy use of RDDs and UDFs. Google says Lightning Engine's native query execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NVIDIA's FAQ lists what its plugin does not accelerate: Dataset operations with custom classes or types, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs. Pandas UDFs are a partial case. Spark runs them with Arrow transferring data to Pandas, which is vectorized on the Python side, but the function still runs in a Python worker, outside any native engine. A job that depends on them keeps that stage on CPU. This is the first check for your own code: count how much of your runtime sits in stages built from UDFs and RDD operations. The guide on [operator coverage and fallback](/guides/operator-coverage-and-fallback/) shows how to find them. ## Fallback: what happens when an operator is not covered All the plugins and engines listed here run unsupported work on the stock Spark engine instead of failing the job. The documented mechanics differ. Apache Auron describes fine-grained fallback and records per-operator reasons in an Auron tab in the Spark UI. Apache DataFusion Comet states that if a query stage contains an operator it does not support, that stage falls back to regular Spark execution, and results are unaffected. Apache Gluten converts between columnar and row formats at each boundary between native and JVM operators. Databricks says Photon falls back transparently. Flarion says operations it does not accelerate run on Spark automatically. Microsoft says Fabric's native execution engine falls back to JVM Spark. Speedata's page says UDFs and other SQL elements that cannot be accelerated are delegated to the CPU. Each fallback has a cost: the conversion between formats, plus the lost speedup on that stage. A job that falls back in the middle of a plan can run slower than the baseline. None of the projects publishes a figure for that cost on a typical job: none found on 2026-11-10. ## How each option defines it | Option (alphabetical) | What changes, per its documentation | | --- | --- | | Apache Auron | Jar built from source (Rust nightly and JDK), spark.auron.enable, a session extension and a shuffle manager; memory moved from off-heap to executor overhead | | Apache DataFusion Comet | spark.plugins, a shuffle manager and off-heap settings; some expressions are off unless allowIncompatible is set | | Apache Gluten with Velox | spark.plugins=org.apache.gluten.GlutenPlugin, off-heap size, ColumnarShuffleManager and a bundle jar loaded ahead of Spark's jars; officially supported Spark versions 3.4.4, 3.5.5, 4.0.2 and 4.1.1 | | Databricks Photon | A cluster setting (runtime_engine PHOTON in the API); runs only on Databricks Runtime; DBU rate differs from the same instance without Photon | | DataPelago | Gluten plugin class, subscription ID and license bucket, off-heap size, columnar shuffle manager; Docker image on EKS or a Dataproc/GKE deployment | | DualBird | Per DualBird: switch the EC2 instance type to F2 and add the DualBird Spark plugin; DualBird says no code changes | | Flarion | One JAR in the existing cluster, installed through an init script, bootstrap action, Helm chart or operator; per Flarion, one config change to install and one to turn off | | Google Lightning Engine | Cluster created with --engine=lightning on image 2.3.3 or later (or 3.0.2 or later); premium rate of $0.089 per DCU-hour versus $0.06 standard in the us-central1 example | | Microsoft Fabric native execution engine | Toggle in an environment or spark.native.enabled true; Fabric only | | NVIDIA cuDF plugin for Apache Spark | Plugin enabled by a configuration setting and GPU instances; Spark 3.3.x to 4.2.0 across releases | | Speedata | spark.plugins=com.speedata.spark.DashPlugin, spark.speedata.apus=1 and a PCIe card in a physical server or a Speedata-hosted AWS service | Dates read 2026-11-10. The Change Budget page lists the same changes as code, cluster, cloud and vendor. ## Where DualBird's definition sits DualBird says its plugin needs no code changes and that "the majority of common Spark performance configurations do not exist when using the solution". Its own engineer states that the FPGA path accelerates a defined set of operators (sort, join, aggregation and others) and that workloads dominated by Python UDFs or unsupported operators fall back to the CPU. Those are DualBird's statements, and DualBird does not publish a supported-operator list. DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. ## Sources - NVIDIA cuDF plugin for Apache Spark: FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - Databricks documentation: What is Photon: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Google Cloud: Lightning Engine for Managed Service for Apache Spark: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Google Cloud: Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Apache Spark documentation: Apache Arrow in PySpark: https://spark.apache.org/docs/latest/api/python/tutorial/sql/arrow_pandas.html (read 10 Nov 2026) - Apache Gluten: getting started: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - Apache Auron: https://auron.apache.org/ (read 10 Nov 2026) - Microsoft Learn: Native execution engine for Fabric Spark: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Flarion FAQs: https://www.flarion.io/faqs (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/what-no-code-change-means/ --- # How to read a vendor Spark benchmark Guide, read 10 Nov 2026 **The short answer.** Seven checks turn a vendor speedup claim into something comparable: who ran it, whether the workload is TPC-DS or real jobs, scale factor, hardware, baseline version, cost basis and published method. Comet's own page shows one engine at 2.0x or 2.51x depending on the Spark baseline. ## Check 1: who ran it A result run by the vendor, a cloud provider, the vendor and a cloud provider together, or a third party are four different kinds of evidence. The Spark Accelerator Ledger on this site records who ran each entry and whether that party sells the product. Nothing is averaged across them. Examples from the published record, read 2026-11-10: AWS and NVIDIA ran the 3 TB GPU test on Amazon EMR on EKS together, and NVIDIA published a separate comparison against open-source Spark on EC2. The Apache Auron and Apache DataFusion Comet projects run their own tests on their own published tooling. Databricks ran its audited 100 TB TPC-DS record on Databricks SQL. Every number DualBird has published so far is DualBird's own. Flarion ran a six-engine comparison on its own page and discloses that its own engine is slower than Databricks Photon in both scales. ## Check 2: TPC-DS and TPC-H derived versus real workloads The TPC benchmarks are fixed query sets. An audited TPC result follows the council's rules and comes with a full report of the run. A "TPC-DS derived" or "TPC-H-like" run does not, and the labels say so. NVIDIA's NDS repository states that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. Google's Lightning Engine launch post says the same of its TPC-H-derived queries. Real workloads are what you run, but a customer tile with no workload attached cannot be checked. DataPelago's site carries customer testimonials with 50% cost reductions. Speedata's home page shows 62.7x on production Spark workloads with 37 servers consolidated to 3. Neither names a baseline, data size or method. Treat them as claims, not measurements. ## Checks 3 to 5: scale factor, hardware and baseline **Scale factor.** SF100 is about 100 GB of raw data, SF1000 about 1 TB. Flarion's comparison runs both on one i4i.8xlarge: Spark 3.5.5 took 1,135 seconds at SF100 and 135.6 minutes at SF1000, so the scale changes the story. Query counts differ too: the AWS and NVIDIA test ran 103 queries, the AWS sample harness runs 104. **Hardware.** Ask for instance type, count, storage and region. The Auron benchmark page publishes the Spark settings and the tooling but not the node type or count. Comet's results page names the r6i family on EKS but not the instance size. The Gluten homepage reports 3.34x on TPC-H and 3.02x on TPC-DS at 3 TB, on Intel Xeon Platinum 8592+ with Spark 3.3.1, without cluster size or scripts. **Baseline version and settings.** This is the check that moves results most. On the same tables and settings, Comet's results page reports Spark 4.2.0 at 994 seconds, Comet 1.2.0 at 488 seconds (2.0x), and Spark 4.1.3 at 1,336 seconds with Comet 1.2.0 at 532 seconds (2.51x). The project explains the gap: Spark 4.1.3 is slower mainly because of the default S3A input stream. Spark 4.2.0 with the classic stream took 1,294 seconds. A 2.0x and a 2.51x claim come from the same engine and the same page. Also check what the baseline was tuned like: Auron's run gave plain Spark 20g executor memory with 4096 overhead and its own engine 8g with 16384 overhead. ## Checks 6 and 7: cost basis and published method A speedup is not a cost result. AWS's GPU post lists $26.35 per hour for the 8-node g7.4xlarge cluster against $10.43 to $10.99 for the CPU clusters, and arrives at $2.06 per run against $2.93 to $3.18 using On-Demand us-east-1 prices as of May 2026. It leaves out the EKS control-plane fee, EBS, S3 and the driver pod. Photon instances consume DBUs at a different rate than the same instance without Photon, so Databricks's cost depends on the multiplier on its pricing page. NVIDIA's own overview compares a different CPU run and states that the stacks differ. Check whether a quoted saving includes license fees: DualBird publishes no plugin price, so a reader cannot tell whether its 55 to 85% lower EC2 cost figure includes one (DualBird's figure, from its Iceberg compaction post). A published method means scripts, configuration and data generation that you could rerun. AWS publishes an emr-spark-benchmark repository with commands but no results. Auron publishes the datagen tool and runner in a public repository. Comet publishes benchmark scripts and every non-default setting. DualBird's compaction methodology is in a download behind a form, and its enrichment tile (110 GB, 6x to 18x) gives no method at all. Flarion lists its caps and partition counts and says q72 is counted at vanilla Spark's time for Gluten and Comet. ## A short worksheet Where DualBird's benchmarks are described on this site, the same questions are asked. DualBird is available on AWS F2 instances only, not on Azure, Google Cloud or Databricks-managed clusters, and every published DualBird benchmark is DualBird's own. | Question | Where the answer should be | | --- | --- | | Who ran it and do they sell the product? | Byline or repository owner | | TPC-DS, TPC-H, derived, or your kind of job? | Workload line and the footnotes | | Scale factor and query count? | Results table or config | | Instance type, count, region? | Hardware paragraph | | Baseline Spark version and its settings? | Config files in the repository | | Cost per run or per job, with which prices and date? | Cost table | | Can you rerun it? | Scripts linked from the page | Entries in the Spark Accelerator Ledger are recorded in this order. ## Sources - Apache DataFusion Comet: TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - AWS Big Data Blog: GPU-accelerated Apache Spark on Amazon EMR with G7 instances: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA NDS benchmark README: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - Apache Auron: benchmarks: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - Flarion benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026) - AWS samples: emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Google Cloud blog: Lightning Engine for Apache Spark: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - DualBird: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/reading-a-vendor-benchmark/ --- # GPU, FPGA and custom silicon for Spark, in plain words Guide, read 10 Nov 2026 **The short answer.** Three kinds of hardware are sold to speed up Spark: FPGAs (DualBird on AWS F2), GPUs (NVIDIA's cuDF plugin) and custom chips (Speedata's C200 APU on a PCIe Gen 5 card). Each takes over a set of SQL operators and leaves the rest to the CPU. ## What offloading means A Spark executor is a JVM process on CPU cores. Hardware acceleration moves the heavy, repetitive parts of SQL and DataFrame operators (scan decode, filter, project, sort, join, aggregation) to a different chip that does that kind of arithmetic on many values at once. The plan, scheduling and fault tolerance stay in Spark. DualBird's own description says the same for its product: Spark still handles planning, scheduling and fault tolerance. Three consequences follow for the person who owns the bill. The accelerator only helps the work it covers. The data has to travel to the chip and back, so the offloaded stage has to be large enough to cover that trip. And the hardware is something you rent, buy or move to, which is why these options appear in the hardware row of the Change Budget. ## FPGAs: DualBird on AWS F2 An FPGA is a chip whose logic can be reconfigured after manufacture, so a vendor can build a pipeline for sort, join and aggregation in hardware. AWS describes F2 as its second-generation FPGA instance. The AWS accelerated-computing table lists three sizes: f2.6xlarge with 24 vCPU, 256 GiB of memory and one AMD Virtex UltraScale+ HBM VU47P FPGA; f2.12xlarge with 48 vCPU, 512 GiB and two FPGAs; and f2.48xlarge with 192 vCPU, 2,048 GiB and eight FPGAs. DualBird says its Spark plugin runs on these instances: you change the EC2 instance type to F2 and add the plugin. A DualBird engineer says Python UDF-heavy work and unsupported operators fall back to the CPU and see less benefit for now. DualBird says 10x to 30x faster processing and 50% to 90% lower cost on its home page and 10x to 100x in its meta description; the two ranges differ on its own site. Its one worked benchmark is a 100 GB sort-based Iceberg compaction it ran itself on synthetic data (about 370 million rows). DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. ## GPUs: the NVIDIA cuDF plugin for Apache Spark A GPU has thousands of small cores built for the same operation across a column of values. NVIDIA's open-source cuDF plugin (also called the RAPIDS Accelerator for Apache Spark) swaps in GPU implementations of supported SQL and DataFrame operators. NVIDIA says the job still completes when it meets an operator or UDF the plugin cannot run, because that part runs on the CPU. Hardware needed, from NVIDIA's download page: GPUs of the Volta, Turing, Ampere, Ada Lovelace, Hopper or Blackwell generations, Linux with glibc 2.28 or newer, and an NVIDIA driver R525 or later for CUDA 12 (R580 or later for CUDA 13). AWS lists its GPU families in the accelerated-computing instance docs, including G4dn, G5, G6, G6e, G7 and G7e, plus P4d, P5 and P6 for larger GPUs. The AWS and NVIDIA published test used g7.4xlarge, which has one NVIDIA RTX PRO 4500 Blackwell GPU, 16 vCPU and 64 GB of memory. The plugin is free; per AWS's post you pay for the GPU instances. NVIDIA has an AutoTuner, a Profiling tool and a Qualification tool that reads CPU event logs and predicts which applications are candidates. ## Custom silicon: Speedata's APU A custom chip is designed from the start for analytics queries. Speedata calls its chip an Analytics Processing Unit (APU). The C200 is a PCIe Gen 5 card, built around the Callisto ASIC, so it needs a physical server. Speedata offers a pre-configured 2U server with two C200 cards, servers from Dell and HPE, and a "Speedata Acceleration for AWS" service in which you keep data in S3, sync changes to a Speedata-hosted environment, run there and push results back. Speedata says its Dash software plugs into the Catalyst optimizer and delegates UDFs and other non-accelerable SQL to the CPU. Speedata's home page claims up to 100x acceleration and up to 90% lower TCO with zero code changes. Those are Speedata's claims. The Microsoft SIGMOD 2026 paper on Fabric hardware acceleration benchmarked the C200 on TPC-H queries at SF100 against a single-core SQL Server baseline, not against Spark, per Speedata's own summary of the paper. No price list is published. Not every option fits a single type. DataPelago says its Nucleus engine targets CPU, GPU and FPGA, and publishes operator microbenchmarks against cuDF; its docs for the Spark product configure the Gluten plugin and do not state a GPU requirement. ## Side by side, by what you have to change The guide on [GPU versus FPGA](/guides/gpu-vs-fpga-for-spark/) lists what each side has published, and the [Change Budget](/change-budget/) lists what each option makes a team change. | Hardware (alphabetical by example) | Example | Where it runs | What you add | | --- | --- | --- | --- | | FPGA | DualBird | AWS F2 instances, eight regions per AWS | F2 instance type and the plugin | | GPU | NVIDIA cuDF plugin | Cloud GPU instances or your own GPU servers | GPU instances, plugin jar, memory and concurrency settings | | Custom silicon (APU) | Speedata C200 | Your servers, OEM servers or Speedata-hosted AWS service | PCIe cards or a hosted service; Dash plugin | Facts from AWS, NVIDIA, DualBird and Speedata pages read 2026-11-10. ## Sources - AWS documentation: Amazon EC2 accelerated computing instance types: https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html (read 10 Nov 2026) - Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - AWS documentation: instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: download and requirements: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) - DataPelago technology page: https://www.datapelago.ai/technology (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/hardware-acceleration-plain-words/ --- # What to do when Spark tuning is not enough Guide, read 10 Nov 2026 **The short answer.** Spark performance tuning costs engineer time, not new hardware: adaptive query execution (on by default since Spark 3.2.0), partition sizing, broadcast joins, file sizes and skew settings. This guide lists each documented default and the signs in the Spark UI that settings alone no longer move the bill. ## What tuning changes, and what it needs Tuning changes configuration and file layout. It needs no new plugin, no new instance type and no vendor contract, and each change can be reverted. A tuned job is the baseline a trial compares against. It also changes the answer to the question every accelerator vendor gets: faster than what? A baseline that still runs 200 untuned shuffle partitions over tiny files makes any engine look fast. This guide lists the settings the Apache Spark documentation names, with the defaults it publishes, then says how to tell when you have run out of room. It is not a promise that any setting helps your jobs. ## The checklist, with documented defaults | Area | Setting (default per Spark docs) | What to check | | --- | --- | --- | | Adaptive query execution | spark.sql.adaptive.enabled (true since Spark 3.2.0) | Confirm the setting is on in the cluster configuration; Spark 3.2.0 made it the default | | Shuffle partitions | spark.sql.shuffle.partitions (200); AQE advisory size 64 MB | With AQE on, Spark coalesces small partitions toward the advisory size; set the initial number high enough | | Broadcast joins | spark.sql.autoBroadcastJoinThreshold (10 MB) | Dimension tables a little over the threshold run as sort-merge joins; check the SQL tab for the join Spark chose | | File sizes on read | spark.sql.files.maxPartitionBytes (128 MB); openCostInBytes (4 MB) | Thousands of files under a few MB each make scan time a listing and open problem | | Skew | spark.sql.adaptive.skewJoin.skewedPartitionFactor (5.0); threshold 256 MB | A partition is skewed only if it is above both the factor times the median and the byte threshold | | Serialization | spark.serializer (Java); Kryo recommended by the guide for network-heavy jobs | Kryo needs class registration; the tuning guide says often as much as 10x faster | | Parallelism | Tuning guide: 2 to 3 tasks per CPU core | Tasks as short as 200 ms are efficient per the guide | Defaults read 2026-11-10 from the Apache Spark SQL performance tuning page and the Tuning Spark page. ## Adaptive query execution, partition sizing and broadcast joins AQE re-plans the query at runtime from statistics gathered at each shuffle. It does three things the Spark documentation describes: it coalesces contiguous shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB), it switches a sort-merge join to a broadcast join when a side turns out small, and it splits skewed partitions in a sort-merge join. Spark 3.2.0 turned it on by default, so on a current runtime the work is checking that nobody turned it off and that spark.sql.shuffle.partitions is not set so low that the coalescing has nothing to start from. Amazon EMR documents AQE, dynamic partition pruning, Bloom filter join and optimized join reorder in its performance-optimized runtime; the AQE-related features are on by default. For broadcast joins, the default of 10 MB is conservative for modern executors. Raising it makes more joins skip the shuffle, but the broadcast table must fit in driver and executor memory, and the timeout is 300 seconds by default (spark.sql.broadcastTimeout). A join hint (BROADCAST, MERGE, SHUFFLE_HASH) overrides the choice for one query, with BROADCAST taking priority over the others per the documentation. ## File sizes and skew Scan cost depends on layout. Spark packs files into read partitions up to 128 MB, charging 4 MB of estimated cost per file opened. A table of many 1 MB files costs more to read than a table of the same bytes in 128 MB files. Compaction jobs fix that, and the Iceberg compaction bill is a separate cost for teams that run it daily. The Spark tuning guide also says to raise parallelism when reduce tasks run out of memory because each handles too much, and notes that listing a very large number of input directories against an object store such as S3 can take a long time. Skew shows up as one task far longer than its siblings. AQE's skew handling only splits partitions above 256 MB that are also more than 5.0 times the median partition size. A skewed key below those thresholds, or skew in a group-by instead of a join, stays. The fixes then are salting a key or filtering the hot value out and handling it separately, which are code changes. ## How to tell tuning is not enough Look for these signs in the Spark UI after the settings above are applied. The stage with the most task time is dominated by CPU, not by waiting on a shuffle fetch or a single straggler. Spill is small, and GC time is a small share of task time. Input size and file counts are reasonable. If those hold and the job is still slow or the bill still rising, the remaining cost is the engine's own execution speed, which is what the options on this site change. The [two-week trial](/guides/trialing-an-accelerator/) guide uses the tuned job as its baseline. The share of [SQL and DataFrame work](/guides/what-no-code-change-means/) against UDF and RDD code matters, because those stages stay on the CPU in every plugin documented here. ## What DualBird says about tuning DualBird claims that tens of Spark settings (executor sizing, CPU-to-memory ratios, parallelism) become irrelevant on its path, that disk spills effectively disappear, and that cluster sizing reduces to the number of F2 nodes. A later post in its series says skew still costs time, but less, and needs no repartitioning or salting. Those are DualBird's claims and it publishes no independent test of them. Those claims hold only on AWS F2 instances, which leaves out Azure, Google Cloud and Databricks-managed clusters, and every DualBird benchmark so far is its own. ## Sources - Apache Spark documentation: SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026) - Apache Spark documentation: Tuning Spark: https://spark.apache.org/docs/latest/tuning.html (read 10 Nov 2026) - Apache Spark documentation: Configuration: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) - Amazon EMR documentation: Spark performance features: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - DualBird product page: https://dualbird.io/product (read 10 Nov 2026) - DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/when-tuning-is-cheaper/ --- # Native Spark execution engines, explained Guide, read 10 Nov 2026 **The short answer.** A native execution engine replaces the JVM operators in a Spark plan with compiled C++ or Rust code that works on columns of values. Apache Auron and Apache DataFusion (Rust), Databricks Photon and Velox (C++, from Meta) all do this, and all fall back to Spark for operators they do not cover. ## What changes inside the executor In stock Spark, a stage runs as generated JVM code, with rows held as Java objects or off-heap Tungsten buffers and memory managed by the garbage collector. A native engine keeps Catalyst as the planner, then hands a whole subtree of the physical plan to a compiled library. The library processes columns in batches, so one function call handles thousands of values and the CPU can use vector instructions. Data stays in columnar buffers outside the JVM heap. The cost is a boundary. Where a native subtree meets a JVM operator, data is converted between row and columnar formats. Each boundary adds a conversion, so a plan fragmented by unsupported operators pays that cost more often. The Comet tuning guide has a page on reducing row and columnar conversions for this reason. ## The engines and their building blocks **Velox** is a C++ library created by Meta and developed with IBM, Intel, Voltron Data, Microsoft, ByteDance and others. Its README says it is not meant to be used directly by end users: it takes an optimized plan and runs it. Its components include an Arrow-compatible vector layout with Flat, Dictionary, Constant and Sequence/RLE encodings, a vectorized expression evaluator, scalar, aggregate and window functions following Presto and Spark semantics, and scan, join, sort, shuffle and write operators. Velox reads ORC/DWRF, Parquet and Nimble files from S3, HDFS, GCS and ABFS. **Apache Auron** (originally Blaze) is a Spark plugin with a native engine written in Rust, built on DataFusion and Arrow. You build the jar yourself with Rust nightly and a JDK. **Apache DataFusion Comet** is a Spark plugin built on Apache DataFusion, a Rust query engine on Apache Arrow. Comet is Apache-2.0 licensed under the Apache Software Foundation and its Maven jars bundle native libraries for Linux on amd64 and arm64. **Apache Gluten** is the middle layer that offloads Spark SQL execution to a native backend, and Velox is the backend most documented. Microsoft Fabric's native execution engine and Google's Lightning Engine are listed as users on Gluten's Powered By page, and DataPelago's documentation configures the Gluten plugin class. **Databricks Photon** is a proprietary C++ vectorized engine inside Databricks Runtime, described in a SIGMOD 2022 paper. It is not available outside Databricks compute. **Flarion** says its engine is built on Apache Arrow and DataFusion and is proprietary. **Microsoft Fabric native execution engine** is Microsoft's packaging of Velox and Gluten for Fabric Spark runtimes 1.3 (Spark 3.5) and 2.0 (Spark 4.1). ## Operator coverage and fallback Each project documents what it does not cover. Gluten's Velox backend page lists known differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 and so have no lookaround, file scans fully support only Parquet and partly ORC, and JSON functions differ for single-quoted strings. Comet's documentation says some expressions have native implementations that differ from Spark and stay off unless allowIncompatible is set, and that Iceberg writes are off by default. Fabric lists decimal-to-float casting, timezone, round(), map() duplicate keys and collect_list ordering as differences on Runtime 1.3. Photon does not support UDFs, RDD or Dataset APIs, or stateful streaming. The guide on [operator coverage and fallback](/guides/operator-coverage-and-fallback/) shows where each project says to look for the list. ## Spark versions, memory and where each one runs All five have the same shape: your code is unchanged, your configuration and runtime are not. Speedups published for them are listed, with who ran each test, in the [Spark Accelerator Ledger](/ledger/). | Engine (alphabetical) | Spark versions documented | Where it runs | Memory note | | --- | --- | --- | --- | | Apache Auron | 3.0 to 4.2 per its build script | Spark on YARN in its benchmark; no EMR, EKS or Databricks support documented | Off-heap disabled in the quick start; memory moved into executor overhead | | Apache DataFusion Comet | 3.5.9, 4.0.4, 4.1.3, 4.2.0; JDK 17 or later | Own Spark clusters; benchmarked on EKS; managed forks may not fully work | Off-heap pool and executor overhead must be set | | Apache Gluten with Velox | Official: 3.4.4, 3.5.5, 4.0.2, 4.1.1 | Own clusters; Ubuntu 20.04/22.04 or CentOS 8 | Off-heap size must be enabled and set | | Databricks Photon | Databricks Runtime version | Databricks on AWS, Azure and GCP | Managed by Databricks | | Microsoft Fabric native execution engine | Runtime 1.3 and 2.0 | Fabric (Azure) only | Managed by Fabric | Read 2026-11-10 from each project's documentation. Using an unofficial Spark version with Gluten can throw NoSuchMethodError at runtime, per its docs. ## What the native projects have published The Apache Gluten homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: 3.34x overall on TPC-H (up to 23.45x on a single query) and 3.02x on TPC-DS (up to 13.75x on a single query). It links no method, and the hardware is from Intel, a contributor. The Apache DataFusion Comet project publishes TPC-DS-derived results at scale factor 1000 on EKS with its scripts and every non-default setting. Apache Auron says about 2x faster and about 50% fewer cluster resources against Spark 3.5 on a TPC-DS 1 TB run it describes end to end, with results shown as a chart. Databricks publishes a 3x average and a more-than-10x maximum for Photon against its own earlier runtime. These numbers come from different baselines, scales, hardware and Spark versions. Do not subtract one from another. The guide on [reading a vendor benchmark](/guides/reading-a-vendor-benchmark/) lists the checks that make two results comparable. ## Sources - Velox README: https://raw.githubusercontent.com/facebookincubator/velox/main/README.md (read 10 Nov 2026) - Apache Gluten: Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Gluten: Powered By: https://gluten.apache.org/poweredby/ (read 10 Nov 2026) - Apache DataFusion Comet: compatibility guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Apache DataFusion Comet: installation: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache Auron: https://auron.apache.org/ (read 10 Nov 2026) - Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - Microsoft Learn: Native execution engine for Fabric Spark: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Flarion about page: https://www.flarion.io/about-us (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/native-engines-explained/ --- # GPU vs FPGA for Spark: the published facts Guide, read 10 Nov 2026 **The short answer.** DualBird says its plugin runs Spark on AWS F2 FPGA instances, and NVIDIA's cuDF plugin runs it on GPUs. Published so far: DualBird's own 100 GB Iceberg compaction test, and a 3 TB test by AWS and NVIDIA at 281 seconds on eight GPU nodes. Both lists are below, with no pick. ## What is being compared This page sets the public record for one GPU option and one FPGA option side by side. The GPU option is the NVIDIA cuDF plugin for Apache Spark (the RAPIDS Accelerator). The FPGA option is DualBird's Spark plugin on Amazon EC2 F2 instances. The tests were run on different workloads, scales and baselines by different parties, so the numbers below are not a head-to-head. They are what each side has published, in one place. Neither technology is judged here, and nothing is derived from the figures. The plain-words explanation of both is in the guide on [hardware acceleration](/guides/hardware-acceleration-plain-words/). ## What NVIDIA and its partners have published The main test is by AWS with NVIDIA on Amazon EMR on EKS in us-east-1. TPC-DS at 3 TB, 103 queries, median of five iterations, Spark 3.5.6 and cuDF plugin 26.04.2. Eight g7.4xlarge nodes (one NVIDIA RTX PRO 4500 Blackwell GPU, 16 vCPU and 64 GB each) took 281 seconds. Comparable eight-node CPU clusters took 1,010 to 1,043 seconds, which AWS labels a 3.7x speedup. AWS reports $2.06 per run on the GPU cluster and $2.93 to $3.18 on the CPU clusters, with the GPU cluster at $26.35 per hour (eight instances at $3.042 plus $2.01 for EMR on EKS) and the CPU clusters at $10.43 to $10.99 per hour. AWS leaves the EKS control-plane fee, EBS, S3 and the driver pod out of the cost. NVIDIA's overview page adds a comparison against separate NDS runs on EC2 with open-source Spark 3.5.6: 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with reported cost per run 77% and 72% lower. NVIDIA itself notes the stacks differ. NVIDIA publishes its NDS automation in a public repository and states NDS is derived from TPC-DS. Coverage: the DataFrame API and a limited set of SQL and DataFrame operations; not RDDs, Dataset lambdas or GraphX. Anything the plugin does not cover, UDFs included, runs on the CPU. Releases support Spark 3.3.x to 4.2.0, and Databricks (14.3 ML LTS and 17.3 ML LTS) and Dataproc runtimes are listed. Amazon EMR on EKS supports the plugin from release 6.9.0, with a "-spark-rapids" variant that ships it. Independent benchmark: none found on 2026-11-10. ## What DualBird has published, labelled as DualBird's DualBird says its plugin gives 10x to 30x faster processing and 50% to 90% lower costs with no changes required. That is the range in its page text; its meta description says 10x to 100x. Its home page shows a 110 GB enrichment pipeline (join denormalization) with 50% to 85% lower EC2 cost and a 6x to 18x higher performance ceiling, with no hardware, baseline, date or method given. Its detailed test is a sort-based Iceberg compaction on 100 GB of synthetic data (about 370 million rows, 64 columns, ZSTD level 3), compared with vanilla Spark and an unnamed C++ accelerated Spark. DualBird says roughly 12x to 20x faster single-task execution, 55% to 85% lower EC2 cost, a practical performance ceiling about 6x higher, and about $2 per terabyte compacted against about $6 per terabyte rising past $13 for CPU clusters. DualBird's page describes the method only in summary; the full methodology is in a download behind a form. Instance types and counts are not stated in the public post. Hardware: AWS lists three F2 sizes, f2.6xlarge (one FPGA, 24 vCPU, 256 GiB), f2.12xlarge (two FPGAs, 48 vCPU, 512 GiB) and f2.48xlarge (eight FPGAs, 192 vCPU, 2,048 GiB). DualBird's engineer says a defined set of operators is accelerated (sort, join, aggregation and others) and Python UDF-heavy work falls back to CPU. No third-party benchmark: none found on 2026-11-10. DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. ## Side by side DataPelago, which lists CPU, GPU and FPGA as targets for its Nucleus engine, published operator microbenchmarks against cuDF 25.04.00 on one H100: up to 10.5x on project and 10.1x on filter, 4.3x on aggregate, as DataPelago's own figures. They are not end-to-end Spark jobs. What to check next depends on your own cloud, regions and job mix. The [trial guide](/guides/trialing-an-accelerator/) covers how to measure it. | Fact | DualBird (FPGA) | NVIDIA cuDF plugin (GPU) | | --- | --- | --- | | Hardware | AWS F2: AMD Virtex UltraScale+ VU47P FPGAs | NVIDIA GPUs, Volta through Blackwell generations | | Where it runs | AWS EC2 F2 only, in the customer's AWS account | EMR on EKS, EMR on EC2, Databricks, Dataproc, Kubernetes, on-premises GPU servers | | Software cost | No price published | Free; pay for GPU instances | | Largest published scale | 110 GB (enrichment tile) and 100 GB (compaction), both DualBird's | 3 TB, run by AWS with NVIDIA | | Who ran published tests | DualBird | AWS with NVIDIA; NVIDIA | | Method published | Compaction method behind a sign-up form | AWS cluster, config, release and prices in the post; NDS scripts public | | Spark versions listed | None found on 2026-11-10 | 3.3.x to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3, 4.2.0 (release 26.10.0) | | Fallback | CPU for unsupported operators and Python UDFs, per DualBird | CPU for unsupported operators and UDFs, per NVIDIA | Facts read 2026-11-10. Vendors are listed alphabetically. ## Sources - AWS Big Data Blog: GPU-accelerated Apache Spark with Amazon EMR and NVIDIA on G7 instances: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - NVIDIA cuDF plugin for Apache Spark: download: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - DualBird: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - AWS documentation: Amazon EC2 accelerated computing instance types: https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html (read 10 Nov 2026) - AWS documentation: instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DataPelago: Nucleus versus NVIDIA cuDF: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/gpu-vs-fpga-for-spark/ --- # Spark acceleration options on Amazon EMR and EKS Guide, read 10 Nov 2026 **The short answer.** On Amazon EMR and EKS, the documented options are the EMR runtime itself, Apache DataFusion Comet (benchmarked on EKS), DataPelago on EKS, DualBird on F2 instances, Flarion and NVIDIA's cuDF plugin (EMR on EKS from release 6.9.0). Photon, Fabric and Lightning Engine do not run there. ## What counts as running on EMR or EKS Amazon EMR runs Spark three ways: EMR on EC2, EMR on EKS and EMR Serverless. EMR on EKS runs Spark pods on your Kubernetes cluster, with EMR images. A team on open-source Spark on EKS without EMR is a fourth case, and some projects document only that. The list below separates what each project documents from what it does not, and says none found on 2026-11-10 where a page is silent. ## The options, in alphabetical order **Amazon EMR Spark runtime.** AWS says the performance-optimized runtime runs up to 4.5x faster than open-source equivalents and gives 2.7x faster Iceberg writes, with no TPC-DS scale or cluster stated on the page. It is described as 100% API compatible with open-source Spark and proprietary to AWS, a tuned build of Spark rather than a native replacement. It is the only option here that needs no plugin, but jobs must run on EMR. AWS publishes the emr-spark-benchmark repository (3 TB TPC-DS, 104 queries, three iterations) with commands for open-source Spark, EMR on EC2, EMR Serverless and a link to EMR on EKS, and a companion emr-on-eks-benchmark repository. It does not publish results there. **Apache Auron.** No EMR, EKS or Databricks support is documented. Its benchmark ran on YARN. The jar is built from source with Rust nightly. **Apache DataFusion Comet.** Its published TPC-DS-derived results at scale factor 1000 ran on an EKS cluster using r6i instances with EBS and S3 data. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and its docs warn it may not fully work with cloud-provider Spark forks, so EMR's runtime is not documented as supported. **Apache Gluten with Velox.** Its documentation does not mention EMR, EKS or Databricks support. Official Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1; docs have pages for S3 storage. **DataPelago.** On AWS, DPA-S is available for open-source Spark on EKS as a Docker container in ECR (Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22). It needs a subscription ID and a license bucket tied to registered AWS account IDs, and is billed through the AWS marketplace pay-as-you-go. EMR is not mentioned on its AWS page. **DualBird.** DualBird says its plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS, and that onboarding is switching the EC2 instance type to F2 and adding its plugin. It lists no Spark version numbers. **Flarion.** Lists Amazon EMR (installed with a bootstrap action) and Spark on Kubernetes (Helm chart or Spark operator) among its platforms. It runs on the CPU fleet already in use. A supported-version list is not published. Pricing is $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP and $0.012 for fully managed Spark including EMR. **NVIDIA cuDF plugin for Apache Spark.** Amazon EMR on EKS supports the plugin from release 6.9.0, and the "-spark-rapids" release variant ships it pre-installed. NVIDIA's EMR guide lists the plugin jar each EMR release ships: EMR 7.13 and 7.12 carry Spark 3.5.6 with rapids-4-spark 25.08.0, EMR 7.0 carries Spark 3.5.0 with 23.10.0, and EMR 6.10 carries Spark 3.3.1 with 22.12.0. Needs GPU instances, such as the g7.4xlarge used in AWS's test. **Speedata.** Not an instance family: the APU is a PCIe card, and the AWS option is a Speedata-hosted environment where the customer keeps data in S3 and results are pushed back. Speedata does not run inside the customer's EC2 fleet, and EMR and EKS support are not stated on the pages read. ## Not available on EMR or EKS Databricks Photon is documented only as part of Databricks compute. Microsoft Fabric's native execution engine runs only inside Fabric on Azure. Google's Lightning Engine is a Google Cloud service. A team that owns an EMR or EKS bill cannot add them without moving platforms. ## What each one needs, side by side DualBird's limits in one line: AWS F2 instances only, so not Azure, Google Cloud or Databricks-managed clusters, and every benchmark it has published is its own. | Option | Instance or hardware | Spark versions documented | Billing note | | --- | --- | --- | --- | | Amazon EMR Spark runtime | Standard EMR instances | Per EMR release; Spark 4.0 and Iceberg v3 on the features page | EMR price added to EC2 and EBS; EMR on EKS per vCPU-hour and GB-hour requested, one-minute minimum | | Apache DataFusion Comet | Any x86 or arm64 Linux; r6i in its benchmark | 3.5.9, 4.0.4, 4.1.3, 4.2.0 | Open source | | DataPelago | EKS pods; hardware requirement not confirmed | 3.5.1 on AWS | Marketplace pay-as-you-go; unit price not published | | DualBird | AWS F2 (f2.6xlarge to f2.48xlarge) | None found on 2026-11-10 | No plugin price published | | Flarion | Existing CPU fleet | 3.5.5 in its benchmark; list not published | Per accelerated CPU hour | | NVIDIA cuDF plugin | NVIDIA GPU instances, for example g7.4xlarge | 3.3.x to 4.2.0 across releases | Software free; GPU instance and EMR on EKS charges apply | Alphabetical; Auron, Gluten and Speedata omitted because no EMR or EKS support is documented. Read 2026-11-10. ## Questions to settle before a trial Which EMR release label and Spark version you run today decides which options are even installable. Whether your jobs run on EC2 or EKS decides how the plugin reaches the executors: a bootstrap action, a custom image in ECR, or a release label. Which region you run in decides whether F2 or a given GPU family is offered. And whether the claimed saving includes the instance and EMR surcharge is something the vendor must state. The [Change Budget](/change-budget/) records these as the cloud and cluster rows, and the guide on [trialing an accelerator](/guides/trialing-an-accelerator/) turns them into a plan. ## Sources - Amazon EMR: Apache Spark features: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS samples: emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - NVIDIA cuDF plugin: getting started on AWS EMR: https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/aws-emr.html (read 10 Nov 2026) - AWS Big Data Blog: GPU-accelerated Apache Spark on Amazon EMR: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Apache DataFusion Comet: installation: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - DataPelago: deploy on AWS: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - Flarion pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/spark-acceleration-on-emr-and-eks/ --- # Operator coverage and fallback in Spark accelerators Guide, read 10 Nov 2026 **The short answer.** Before any trial, check which operators your jobs use against each project's supported-operator list. Comet marks each Spark operator Supported, Supported (caveats) or Planned, Gluten lists known fallbacks such as ANSI mode, and NVIDIA's list is generated against Spark 3.3.0. ## Why coverage decides the result An accelerator speeds up the operators it implements and hands the rest back to Spark. The published speedups come from benchmark queries that mostly use covered operators. Your jobs may use other ones. A job whose long stage contains one unsupported expression can see that stage run on the JVM, with format conversions on both sides. So the first technical question for any option is not how fast, but how much of my plan runs natively. The sources below are the projects' own pages, read 2026-11-10. They differ in granularity, from whole-operator tables to lists of known differences. Where a project publishes no list, this guide says none found. ## Where each project documents coverage | Option (alphabetical) | Where coverage is documented | How to see fallback in a running job | | --- | --- | --- | | Apache Auron | Homepage and docs describe fine-grained fallback for unsupported operators | Auron tab in the Spark UI records per-operator fallback reasons | | Apache DataFusion Comet | Operator support page with a status per Spark physical operator: Supported, Supported (caveats), Planned; compatibility guide for expressions | Plan explain output marks fallback reasons with [COMET: ...] | | Apache Gluten with Velox | Velox backend limitations page; per-format pages for Iceberg, Hudi and Paimon | Columnar-to-row and row-to-columnar conversion nodes in the plan; troubleshooting page | | Databricks Photon | Photon documentation lists unsupported items: UDFs, RDD and Dataset APIs, stateful streaming | Spark UI plan shows which operators ran in Photon | | DataPelago | Docs list limitations: ANSI mode and case-sensitive mode | Not stated on the pages read | | DualBird | DualBird's engineer says a defined set (sort, join, aggregation and others); no operator list: none found on 2026-11-10 | Not stated: none found on 2026-11-10 | | Flarion | FAQ says non-accelerated operations run on Spark | Telemetry marks each stage native or Spark with runtime and memory | | Google Lightning Engine | Docs list unsupported inputs: Byte in ORC and Parquet; Struct, Array and Map in Parquet | Qualification tool checks jobs before migration | | Microsoft Fabric native execution engine | Overview page lists differences on Runtime 1.3 | Fabric Spark Advisor alerts show fallback | | NVIDIA cuDF plugin | Generated supported_ops page by operator and data type; compatibility page | spark.rapids.sql.explain=NOT_ON_GPU logs what stays on CPU | | Speedata | Not published as a list on the pages read | Free Workload Analyzer predicts per-query acceleration from Spark logs | Read 2026-11-10 from each project's pages; see the sources below. ## Reading the lists Comet's operator page says it runs whole subtrees of native operators together, so if a stage contains an operator Comet does not support, that stage falls back to regular Spark execution, and results are unaffected. Each operator can be switched off individually with spark.comet.exec.OPERATOR.enabled=false, which is a way to bisect a wrong result. Expressions whose native implementation differs from Spark are off unless allowIncompatible is set. Gluten's Velox limitations page is a list of differences, not of operators: ANSI mode always falls back, regexp functions use RE2 with no lookaround, scans fully support only Parquet and partly ORC, and Hudi supports reading copy-on-write tables only. DataPelago, which configures the Gluten plugin, adds that native CPU acceleration is not used when ANSI mode is on. Both pages state that case-sensitive mode may give wrong results or is unsupported. NVIDIA's supported-operators page is generated by a tool against Apache Spark 3.3.0, and the page itself says most of the content applies to other versions with variations. It lists expressions by data type. Decimal is a good example: Spark supports precision up to 38 digits, and the plugin supports 128-bit decimals from version 21.12. A job full of wide decimals needs a check on the type, not just the operator name. ## What happens on fallback The job still finishes. That is the documented behavior of every option above. What changes is cost. Comet and Gluten convert between columnar and row formats at each boundary. NVIDIA's FAQ adds that on fallback the job completes while the fallen-back stages are not accelerated. Photon falls back for the remainder of an operation. Lightning Engine's native query execution fails on GPU accelerators, per Google, which is a different failure mode from fallback. No project publishes a measurement of the fallback penalty on a typical job: none found on 2026-11-10. You measure it in your own trial: compare the runtime of fully native stages with stages that fell back and note the conversion nodes in the plan. ## A coverage check in four steps 1. Collect the plans. For your five longest jobs, capture the physical plan from the SQL tab of the Spark UI or from EXPLAIN FORMATTED, and list every operator and function in the long stages. 2. Look them up in the lists above. Mark each as covered, covered with caveats, or not listed. Pay attention to data types, ANSI mode and case sensitivity settings. 3. Run the vendor's own estimator on your logs where one exists: Flarion's free assessment (which Flarion says simulates each job from logs), Google's qualification tool, NVIDIA's Qualification tool or Speedata's Workload Analyzer. Treat their output as a prediction and check it in the trial. 4. Turn on the explain or UI view during the trial and count the stages that fall back. The [trial guide](/guides/trialing-an-accelerator/) puts this in the plan, and the guide on [native engines](/guides/native-engines-explained/) explains the conversion boundary. ## DualBird and coverage DualBird says Spark still plans the job and its FPGA path covers a defined set of operators, with Python UDF-heavy work falling back to CPU, and that coverage is expanding. That is DualBird's own description. That description applies only on AWS F2 instances, not on Azure, Google Cloud or Databricks-managed clusters, and every DualBird benchmark published so far is DualBird's own. ## Sources - Apache DataFusion Comet: Spark operator support: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/operators.md (read 10 Nov 2026) - Apache DataFusion Comet: compatibility guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - NVIDIA cuDF plugin: supported operators: https://raw.githubusercontent.com/NVIDIA/spark-rapids/main/docs/supported_ops.md (read 10 Nov 2026) - NVIDIA cuDF plugin: Qualification tool: https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html (read 10 Nov 2026) - Apache Gluten: Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Apache Auron: https://auron.apache.org/ (read 10 Nov 2026) - Databricks documentation: What is Photon: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - DataPelago: deploy on AWS: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - Google Cloud: Lightning Engine for Managed Service for Apache Spark: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Microsoft Learn: Native execution engine for Fabric Spark: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Flarion FAQs: https://www.flarion.io/faqs (read 10 Nov 2026) - Speedata Workload Analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - DualBird: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/operator-coverage-and-fallback/ --- # How to run a fair two-week trial on your own jobs Guide, read 10 Nov 2026 **The short answer.** A fair trial fixes the baseline, the data and the cluster cost, then measures cost per job from event logs. Spark leaves event logging off by default (spark.eventLog.enabled is false), so turn it on in week one. Published tests use anything from one run to a median of five. ## What the trial has to answer Published benchmarks answer a question about someone else's workload. A trial answers one question about yours: for the same jobs on the same data, what does a finished job cost with the accelerator, and what does it cost without? Everything below is there to keep the two sides comparable. The plan covers 14 days: week one for the baseline and setup, week two for the accelerated runs and the comparison. The trial does not decide anything for you. It produces a table of your own numbers that you can set beside the published ones in the [Spark Accelerator Ledger](/ledger/). ## Week one: baseline, data and logs **Choose the jobs.** Pick five to ten jobs that between them account for most of the bill, including at least one that runs late. Use the stage split from the guide on [where Spark time goes](/guides/where-spark-time-goes/) to see which ones are scan, shuffle or compute heavy. Check each against the options' [definitions of no code change](/guides/what-no-code-change-means/) and the [operator lists](/guides/operator-coverage-and-fallback/). **Tune the baseline first.** The baseline should be the configuration you would run in production today, not the one from last year. Check AQE, shuffle partitions, broadcast thresholds and small files, as listed in the [tuning guide](/guides/when-tuning-is-cheaper/). The Comet project's own page shows why: the same engine looked 2.0x or 2.51x faster depending on which Spark version it was compared with. **Freeze the data.** Run on a fixed snapshot of input tables, or on one day's data replayed, so both sides read the same bytes. A moving table breaks the comparison. **Turn on event logs.** spark.eventLog.enabled defaults to false, so set it to true with a log directory before the first run, and run the Spark History Server so the UI can be rebuilt after the application ends. For long applications the documentation points to rolling event logs (spark.eventLog.rolling.enabled and maxFileSize) to avoid one huge file. Keep the logs for every run, baseline and trial. **Repeat the baseline.** Run each baseline job several times across different days. AWS and NVIDIA report a median of five iterations; the Comet project runs each query twice and reports the mean; Flarion uses two runs at 100 GB and one at 1 TB. Your own variation from noisy neighbors, S3 latency and spot interruptions is the floor under any claimed gain. ## Week two: same cluster cost, then the accelerated runs Decide the unit before you start. The fair comparison is cost per finished job, not seconds. Fix one of two things: the same cluster cost per hour (so a faster job is cheaper) or the same job output at whatever cluster the accelerator needs. Do not run an accelerated job on a bigger cluster and compare runtimes. Record per run: wall-clock time, instance types and counts, hours billed, and any license or service fee. Billing details differ. EMR on EKS charges per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2. On Databricks, Photon instance types carry their own DBU multiplier, so log it per run. DualBird publishes no plugin price, so ask for one in writing before the trial starts. Flarion bills per accelerated CPU hour. Google bills Lightning Engine at a premium rate ($0.089 per DCU-hour against $0.06 standard in the pricing page's us-central1 example). Speedata publishes no price list, so ask it too. For NVIDIA, the AWS post states its method: cost per run is cluster hourly cost times run time, with the excluded items listed. Cost per job = (sum of instance hours times hourly price, plus platform and license fees) for the job's runs. Write the formula in your results sheet and fill it from billing records, not from estimates. ## What to read in the Spark UI and event logs For each run, compare the stage durations, shuffle read and write sizes, spill and GC time. On the SQL tab, count how many operators ran natively and how many fell back. Native engines show the evidence in different places, listed in the guide on operator coverage. A job that shows a large speedup on three stages and none on the fourth is telling you what the fourth contains. Run the vendor's estimator on the baseline logs before the trial if one exists. NVIDIA's Qualification tool analyzes CPU event logs and predicts GPU duration and speedup with an XGBoost model trained on paired CPU and GPU runs. Speedata's Workload Analyzer works from uploaded Spark logs, and Flarion offers a free assessment that simulates each job from logs. Compare their predictions with what you measure. The gap is itself a finding. ## The results sheet and the questions it must answer Add two checks the published tests rarely cover: correctness (compare output row counts and checksums with the baseline; projects list differences, such as regexp semantics or ANSI mode) and operations (what it takes to upgrade Spark, restart executors and debug a failure with the plugin on). Then stop at what you measured. A two-week result on ten jobs is evidence about those ten jobs. Whether it applies to the rest depends on how similar they are, which the stage splits and fallback counts let you judge. | Column | Why it is there | | --- | --- | | Job, input snapshot, Spark version | Proves both sides ran the same thing | | Baseline cost per job (median of N runs) | The number to beat, after tuning | | Trial cost per job (median of N runs) | Same formula, same prices | | Stages native versus fallback | Explains the result and predicts the next job | | Failures, wrong results, config changes needed | The cost of running it in production | | Fees and prices not yet quoted | Items still open, such as a plugin price | One row per job. ## Sources - Apache Spark documentation: Monitoring and instrumentation: https://spark.apache.org/docs/latest/monitoring.html (read 10 Nov 2026) - Apache Spark documentation: Configuration: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) - Apache Spark documentation: Web UI: https://spark.apache.org/docs/latest/web-ui.html (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS Big Data Blog: GPU-accelerated Apache Spark on Amazon EMR: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cuDF plugin: Qualification tool: https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html (read 10 Nov 2026) - Apache DataFusion Comet: TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Google Cloud: Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Flarion pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Speedata Workload Analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/guides/trialing-an-accelerator/ --- # Spark acceleration terms, each with a sourced example Glossary, read 10 Nov 2026 **The short answer.** Short definitions of the terms used across the site, such as Velox, operator fallback, off-heap memory, shuffle, TPC-DS, AWS F2 instances and Iceberg compaction. Each entry gives a plain definition, an example from a published source and a link to the page that states it. ## 35 terms, A to Z - [Adaptive Query Execution](/glossary/adaptive-query-execution/) - [Apache Arrow](/glossary/apache-arrow/) - [APU (Analytics Processing Unit)](/glossary/apu/) - [AWS F2 instance](/glossary/aws-f2-instance/) - [Broadcast hash join](/glossary/broadcast-hash-join/) - [Catalyst](/glossary/catalyst/) - [Columnar batch](/glossary/columnar-batch/) - [Cost per job](/glossary/cost-per-job/) - [DataFusion](/glossary/datafusion/) - [Dynamic partition pruning](/glossary/dynamic-partition-pruning/) - [EMR on EKS](/glossary/emr-on-eks/) - [EMR release label](/glossary/emr-release-label/) - [Executor](/glossary/executor/) - [FPGA](/glossary/fpga/) - [GPU](/glossary/gpu/) - [Iceberg compaction (rewrite_data_files)](/glossary/iceberg-compaction/) - [Off-heap memory](/glossary/off-heap-memory/) - [Operator fallback](/glossary/operator-fallback/) - [Parquet row group](/glossary/parquet-row-group/) - [Partition](/glossary/partition/) - [Scale factor](/glossary/scale-factor/) - [Shuffle](/glossary/shuffle/) - [Skew](/glossary/skew/) - [Sort-merge join](/glossary/sort-merge-join/) - [Spark Operator](/glossary/spark-operator/) - [Spark plugin (spark.plugins)](/glossary/spark-plugin/) - [Speculative execution](/glossary/speculative-execution/) - [Spill](/glossary/spill/) - [Substrait](/glossary/substrait/) - [TPC-DS](/glossary/tpc-ds/) - [TPC-H](/glossary/tpc-h/) - [Tungsten](/glossary/tungsten/) - [Vectorized execution](/glossary/vectorized-execution/) - [Velox](/glossary/velox/) - [Whole-stage code generation](/glossary/whole-stage-code-generation/) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/ --- # Adaptive Query Execution Glossary, read 10 Nov 2026 **The short answer.** Adaptive Query Execution (AQE) lets Spark SQL re-optimize a running query from runtime statistics. The Spark documentation says it has been enabled by default since Apache Spark 3.2.0, controlled by spark.sql.adaptive.enabled. It coalesces small shuffle partitions, splits skewed ones and can switch a sort-merge join to a broadcast join. AQE waits until a shuffle stage finishes, reads the real partition sizes, and then rewrites the rest of the plan. That is why a plan shown by EXPLAIN before the run can differ from the one the Spark UI shows afterward. Native engines that plug into Spark inherit this behavior only for the parts of the plan they leave to Spark. When you compare options, check whether the benchmark baseline ran with AQE on. See also [skew](/glossary/skew/) and [broadcast hash join](/glossary/broadcast-hash-join/). ## A sourced example The Spark configuration page lists spark.sql.adaptive.localShuffleReader.enabled (default true): after AQE turns a sort-merge join into a broadcast-hash join, Spark reads the shuffle files locally instead of over the network. ## Sources - Apache Spark documentation: SQL performance tuning, Adaptive Query Execution: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/adaptive-query-execution/ --- # Broadcast hash join Glossary, read 10 Nov 2026 **The short answer.** A broadcast hash join copies a small table to every executor and joins it against the large table in place, so the large side is never shuffled. Spark broadcasts a table automatically when it is under spark.sql.autoBroadcastJoinThreshold, which defaults to 10485760 bytes (10 MB) and is switched off with -1. Because no shuffle is needed for the big side, this usually costs less than a shuffled join. It fails when the small side is not small, since every executor must hold a full copy in memory. You can force it with the BROADCAST join hint. AQE can also pick it at runtime when a table turns out smaller than the planner estimated. ## A sourced example With the default 10 MB threshold, a 40 MB dimension table is not broadcast automatically. Raising spark.sql.autoBroadcastJoinThreshold above 41943040 bytes, or adding a BROADCAST hint, changes the plan without touching the join logic. ## Sources - Apache Spark documentation: SQL performance tuning, join strategy: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/broadcast-hash-join/ --- # Sort-merge join Glossary, read 10 Nov 2026 **The short answer.** A sort-merge join sorts both sides of a join on the join key and then walks the two sorted streams together. It is Spark's default for large equi-joins, but both sides are shuffled and sorted first, which makes it a common source of long stages. Sort and shuffle are the work that native-engine and hardware vendors say they take over. The Databricks post below shows JVM code generation helping hash join far more than sort-merge join. The MERGE join hint asks Spark for this strategy. See [shuffle](/glossary/shuffle/) and [spill](/glossary/spill/) for the costs around it. ## A sourced example In the Databricks engineering post on Spark 2.0, whole-stage code generation cut hash join cost per row from 115 ns to 4.0 ns, but sort-merge join only moved from 750 ns to 700 ns. ## Sources - Databricks blog: Apache Spark as a Compiler (May 23, 2016): https://www.databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/sort-merge-join/ --- # Shuffle Glossary, read 10 Nov 2026 **The short answer.** A shuffle is Spark's mechanism for redistributing data so that it is grouped differently across partitions. It copies data between executors and machines, which the Spark programming guide calls a complex and costly operation. Joins, group-bys and repartitions trigger shuffles, and spark.sql.shuffle.partitions sets 200 output partitions by default. Shuffle cost has three parts: serializing and writing map output to local disk, moving it over the network, and reading and merging it on the other side. Disk and network are why shuffle-heavy jobs slow down on small or throttled instances. Several accelerators replace parts of this path. Gluten and Comet ship their own columnar shuffle manager, and remote shuffle services such as Celeborn move it off the executors. ## A sourced example The Spark RDD guide uses reduceByKey as its example: all values for one key must sit on the same partition, so Spark reads from every partition and moves the matching rows together. ## Sources - Apache Spark RDD programming guide: Shuffle operations: https://spark.apache.org/docs/latest/rdd-programming-guide.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/shuffle/ --- # Spill Glossary, read 10 Nov 2026 **The short answer.** Spill is what happens when Spark cannot hold shuffle or aggregation data in execution memory and writes it to local disk instead. The Spark web UI reports two figures per stage: shuffle spill (memory), the deserialized size in RAM, and shuffle spill (disk), the serialized size written out. Spill is a symptom, not a setting. It usually points to partitions that are too large, skewed keys, or executors with too little memory per task. Fixing it by hand means changing partition counts and memory ratios. Some vendors say their option reduces or removes spill. Those are vendor claims; read them against the Spark UI numbers from a real job. ## A sourced example The Spark configuration reference lists spark.shuffle.spill.diskWriteBufferSize, default 1024 * 1024 bytes (1 MiB), the buffer used when writing sorted records out to the on-disk spill file. ## Sources - Apache Spark documentation: Web UI, Stages tab: https://spark.apache.org/docs/latest/web-ui.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/spill/ --- # Whole-stage code generation Glossary, read 10 Nov 2026 **The short answer.** Whole-stage code generation fuses a chain of Spark SQL operators into one generated Java function, so rows pass between operators in CPU registers instead of through virtual next() calls. It arrived in Spark 2.0 as part of the second-generation Tungsten engine, described in a May 2016 Databricks engineering post. Before it, Spark 1.x used the Volcano iterator model, where each operator returned one row per call. Fusing the operators removed most of that overhead for filters, aggregates and hash joins. Native engines such as Velox take the idea further by processing batches of columns with vectorized code. See [Tungsten](/glossary/tungsten/) and [vectorized execution](/glossary/vectorized-execution/). ## A sourced example The 2016 post measured single-thread cost per row in Spark 1.6 against 2.0: filter 15 ns to 1.1 ns, sum without group 14 ns to 0.9 ns, Parquet decoding of one int column 120 ns to 13 ns. ## Sources - Databricks blog: Apache Spark as a Compiler (May 23, 2016): https://www.databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/whole-stage-code-generation/ --- # Tungsten Glossary, read 10 Nov 2026 **The short answer.** Tungsten is the Apache Spark project, announced by Databricks on April 28, 2015, to bring Spark closer to bare metal, so hardware rather than JVM overhead sets the limit. It had three initiatives: explicit memory management and binary processing, cache-aware computation, and code generation. Spark 2.0 shipped the second generation in 2016. The Tungsten post argued that Spark workloads were increasingly bound by CPU and memory use rather than disk or network. That is the same argument native-engine and hardware accelerators make today, now applied below the JVM. Tungsten's binary-processing idea is why Spark has an off-heap memory setting. See [off-heap memory](/glossary/off-heap-memory/). ## A sourced example The post says a hash table working directly on binary data with memory managed by Spark would first appear in Spark 1.4, used for DataFrame and SQL aggregations, with sort and join structures planned for Spark 1.5. ## Sources - Databricks blog: Project Tungsten (April 28, 2015): https://www.databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/tungsten/ --- # Catalyst Glossary, read 10 Nov 2026 **The short answer.** Catalyst is the query optimizer inside Spark SQL. It turns a SQL string or DataFrame into a logical plan, rewrites it with rules such as predicate pushdown and column pruning, and picks a physical plan. The Apache Spark site says Catalyst automatically rewrites queries to execute more efficiently. Most native Spark accelerators keep Catalyst in charge of planning and replace only the execution of the chosen physical plan. That is why they can run existing SQL and DataFrame code unchanged. Where an operator has no native version, the plan keeps the Spark operator. See [operator fallback](/glossary/operator-fallback/). ## A sourced example Databricks's Photon documentation says Catalyst still plans the query while Photon replaces the JVM execution layer with a C++ runtime for supported operations. ## Sources - Apache Spark: Spark SQL overview: https://spark.apache.org/sql/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/catalyst/ --- # Vectorized execution Glossary, read 10 Nov 2026 **The short answer.** Vectorized execution processes a batch of values from one column per function call instead of one row per call. That cuts per-row overhead and lets the CPU use SIMD instructions. Spark's Parquet reader is vectorized and reads 4096 rows per batch by default, set by spark.sql.parquet.columnarReaderBatchSize. Spark's Parquet reader hands over columnar batches at the scan. Native engines say they keep data columnar through more of the plan, including shuffle. Batch size is a tradeoff. Larger batches amortize overhead but raise memory use and out-of-memory risk, which the Spark documentation says to choose carefully. ## A sourced example Spark's spark.sql.inMemoryColumnarStorage.batchSize defaults to 10000 rows for cached tables, a different setting from the 4096-row Parquet reader batch, and the two do not need to match. ## Sources - Apache Spark configuration reference: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/vectorized-execution/ --- # Columnar batch Glossary, read 10 Nov 2026 **The short answer.** A columnar batch is a group of rows stored column by column in memory, so a filter reads one contiguous array per column. In Spark the class is org.apache.spark.sql.vectorized.ColumnarBatch, which wraps several ColumnVectors and also offers a row view for code that expects rows. Native accelerators hand data between Spark and C++ in columnar batches. When an operator falls back to Spark, a conversion between columnar batches and rows is needed, and that conversion has a cost. The class is marked DeveloperApi, so its shape can change between Spark releases. That is one reason plugin vendors list supported Spark versions. ## A sourced example Spark's SparkSessionExtensions has an injectColumnar method, documented in the Spark 4.3.0 API, that injects a rule to override columnar execution on an executor. Plugins hook in this way without changing job code. ## Sources - Apache Spark 4.3.0 API: ColumnarBatch: https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/vectorized/ColumnarBatch.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/columnar-batch/ --- # Apache Arrow Glossary, read 10 Nov 2026 **The short answer.** Apache Arrow defines a language-independent columnar memory format for flat and nested data, laid out for analytic work on CPUs and GPUs. Because every language reads the same layout, Arrow supports zero-copy reads without serialization. Engines such as DataFusion are built on it. For Spark acceleration, Arrow matters in two places: it is the in-memory format of DataFusion-based engines such as Comet and Flarion, and it is a common exchange format between the JVM and native code. Arrow is a format and a set of libraries, not a query engine. See [DataFusion](/glossary/datafusion/). ## A sourced example The Arrow columnar format specification reads Version 1.5 in the documentation for Arrow v26.0.1, and describes buffers that are relocatable without pointer swizzling, which allows true zero-copy access in shared memory. ## Sources - Apache Arrow home page: https://arrow.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/apache-arrow/ --- # Velox Glossary, read 10 Nov 2026 **The short answer.** Velox is an open source C++ library of execution components, created by Meta, that takes an optimized query plan as input and performs the computation. It is not a database. Apache Gluten and Microsoft Fabric's native execution engine use it to run Spark plans outside the JVM. The Velox README says it is developed in partnership with IBM/Ahana, Intel, Voltron Data, Microsoft, ByteDance and others. Spark gets Velox through a plugin layer, not by adopting Velox as its engine. Velox covers a long list of operators, but not all of Spark's. The rest fall back; see [operator fallback](/glossary/operator-fallback/). ## A sourced example Microsoft's Fabric documentation says its Native Execution Engine is based on Velox and on Apache Gluten, and that Fabric Runtime 1.3 (Spark 3.5) and Runtime 2.0 (Spark 4.1) are supported. ## Sources - Velox README, GitHub: https://raw.githubusercontent.com/facebookincubator/velox/main/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/velox/ --- # DataFusion Glossary, read 10 Nov 2026 **The short answer.** Apache DataFusion is a query engine written in Rust that uses the Apache Arrow in-memory format. It offers SQL and DataFrame APIs with built-in support for CSV, Parquet, JSON and Avro. It began inside the Arrow project and now lists subprojects including Ballista, Python bindings and Comet. DataFusion is a toolkit for building engines, not a Spark replacement. Comet takes DataFusion's native operators and plugs them into Spark's execution path through the JVM. Flarion, a commercial plugin, also says it is built on Arrow and DataFusion. The two products differ in license and support terms, so read each project's own documents. ## A sourced example The Comet project's TPC-DS run at scale factor 1000 on an EKS cluster reported 994 s on Spark 4.2.0 and 488 s on Comet 1.2.0; the project ran it, so read it as a project number. ## Sources - Apache DataFusion user guide: Introduction: https://datafusion.apache.org/user-guide/introduction.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/datafusion/ --- # Substrait Glossary, read 10 Nov 2026 **The short answer.** Substrait is a cross-language specification for describing compute operations on structured data, with serialized forms so that different systems can pass a query plan to each other. In Spark acceleration it is the hand-off format between the JVM planner and a native engine. The Substrait site describes it as a format designed for interoperability across languages and systems, with a focus on the semantics of each operation. Gluten converts Spark's physical plan into a Substrait plan, then a native backend such as Velox executes it. A Spark function with no Substrait mapping cannot be offloaded and stays on the JVM. ## A sourced example The Apache Gluten homepage describes its architecture in one sentence: Spark physical plans are converted to Substrait and executed by native backends, with results returned through the Spark Columnar API. ## Sources - Substrait home page: https://substrait.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/substrait/ --- # Off-heap memory Glossary, read 10 Nov 2026 **The short answer.** Off-heap memory is memory Spark or a plugin allocates outside the JVM heap. The setting spark.memory.offHeap.size defaults to 0 and must be a positive value when spark.memory.offHeap.enabled is true. Native engines such as Gluten and Comet need an off-heap pool because their C++ or Rust code allocates there. Off-heap use is why native plugins ask for new memory settings. The total container size still has to fit your executor limit, so the JVM heap usually shrinks when you grow the off-heap pool. The Spark documentation notes the setting has no effect on heap usage, so if executors must fit a hard limit you shrink the heap yourself. ## A sourced example Comet's TPC-DS benchmark configuration used spark.memory.offHeap.size=32G with 64G executor memory, 10G overhead and spark.comet.exec.memoryPool.fraction=0.8. ## Sources - Apache Spark configuration reference: memory: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/off-heap-memory/ --- # Operator fallback Glossary, read 10 Nov 2026 **The short answer.** Operator fallback is what an accelerator does when it cannot run a Spark operator or expression: that part of the plan runs on regular Spark, and the rest runs on the accelerator. The job finishes with the same result. The part that fell back runs at plain Spark speed, and rows and columns may be converted at the boundary. Fallback is the main reason one workload speeds up more than another. Jobs dominated by Python UDFs or unsupported expressions spend most of their time in fallback. Each option reports it differently: a marker in the explain output, a tab in the Spark UI, or a setting such as spark.rapids.sql.explain=NOT_ON_GPU. Ask a vendor how to see it before you trust a benchmark. ## A sourced example The Apache Gluten Velox backend limitations page lists known fallbacks: ANSI mode always falls back, and regexp functions use RE2, which has no lookaround, so affected expressions can differ from Spark. ## Sources - Apache Gluten home page: https://gluten.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/operator-fallback/ --- # Spark plugin (spark.plugins) Glossary, read 10 Nov 2026 **The short answer.** A Spark plugin is a class loaded into the driver and executors through the spark.plugins setting. Spark documents the SparkPlugin interface, in the org.apache.spark.api.plugin package, as a DeveloperApi that can be loaded dynamically, with an optional driver-side component and an optional executor-side component. Most accelerators in this comparison install this way: a jar on the classpath plus one or more settings, with no change to the job itself. That is also why Spark version support matters, since DeveloperApi classes can change. A plugin does not by itself change execution. Execution changes come from extra rules the plugin registers, described under [columnar batch](/glossary/columnar-batch/). ## A sourced example Speedata's Dash software is enabled with spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1, using the jar speedata-dash-0.8.1-spark_2.12.jar named on the company's technology page. ## Sources - Apache Spark 4.3.0 API: SparkPlugin: https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/plugin/SparkPlugin.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/spark-plugin/ --- # Executor Glossary, read 10 Nov 2026 **The short answer.** An executor is a process launched for a Spark application on a worker node. It runs tasks and keeps data in memory or on disk across them. Each application has its own executors, which stay up for the whole application, and tasks from different applications run in different JVMs. Executor size is where most manual tuning sits: cores per executor, memory per executor, overhead memory and the number of executors. Accelerators that need a different shape, such as GPU plugins with a concurrent-tasks-per-GPU setting, change these numbers. On EMR on EKS each executor is a Kubernetes pod, billed by requested vCPU and memory. ## A sourced example Comet's published TPC-DS run used 32 executors of 16 cores each, with 64G memory per executor and 512 shuffle partitions, as listed in its benchmark guide. ## Sources - Apache Spark documentation: Cluster mode overview: https://spark.apache.org/docs/latest/cluster-overview.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/executor/ --- # Partition Glossary, read 10 Nov 2026 **The short answer.** A partition is a slice of a dataset that one Spark task processes. For file sources, spark.sql.files.maxPartitionBytes packs up to 128 MB into one partition by default. For shuffles, spark.sql.shuffle.partitions sets 200 partitions by default, and AQE can coalesce them after the stage runs. Too many small partitions waste time on scheduling and file opens. Too few large ones cause spill, skew and idle cores. Accelerators change what a good partition size is. NVIDIA's tuning guide lists concurrent GPU tasks as a setting, and DualBird says only a few tasks run at once on its FPGA, so settings tuned for CPU executors may not carry over. ## A sourced example The RAPIDS Accelerator tuning guide lists input partition size and shuffle partitions as settings to revisit; in AWS's g7.4xlarge test the GPU configuration set concurrentGpuTasks to 3. ## Sources - Apache Spark configuration reference: spark.sql.files.maxPartitionBytes: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/partition/ --- # Skew Glossary, read 10 Nov 2026 **The short answer.** Skew is an uneven spread of data across partitions, so a few tasks do most of the work while others finish early. Spark AQE treats a partition as skewed when it is larger than 256 MB and larger than 5.0 times the median partition size, and then splits it. A skewed stage runs as long as its slowest task, so the cluster sits idle while one executor finishes. Raising the cluster size does not fix it. DualBird says skew costs it less time because few tasks run at once on its hardware. That is a vendor claim. See [AQE](/glossary/adaptive-query-execution/) for Spark's built-in handling. ## A sourced example The two defaults that define a skewed partition are spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes = 256MB and spark.sql.adaptive.skewJoin.skewedPartitionFactor = 5.0; both conditions must hold. ## Sources - Apache Spark configuration reference: adaptive skew join: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/skew/ --- # Speculative execution Glossary, read 10 Nov 2026 **The short answer.** Speculative execution relaunches a slow task on another executor and keeps whichever copy finishes first. It is off by default (spark.speculation=false). When on, a task is a candidate if it is 3 times slower than the median and 90% of the stage's tasks are done. Speculation helps when one node is slow, and does nothing for skew, since a duplicate of an oversized task is still oversized. It also spends extra cores, which matters when you pay per vCPU-hour. Check whether a benchmark baseline had it on. It changes elapsed time and cost without changing the job. ## A sourced example The Spark defaults are spark.speculation.multiplier=3, spark.speculation.quantile=0.9 and spark.speculation.minTaskRuntime=100ms, with a check interval of 100ms. ## Sources - Apache Spark configuration reference: speculation: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/speculative-execution/ --- # TPC-DS Glossary, read 10 Nov 2026 **The short answer.** TPC-DS is a decision-support benchmark from the Transaction Processing Performance Council, for ad hoc analytic queries over a retail-style data model. Most Spark accelerator results are TPC-DS derived runs set up by the vendor, which are not audited TPC results and cannot be compared with them. Vendors often write TPC-DS-derived, TPC-DS-like or NDS to signal they used their own scripts. Read the data size, the number of queries run (published runs use 103 and 104), the baseline and who ran it. Audited results come with a published TPC report; derived runs usually do not. See [scale factor](/glossary/scale-factor/). ## A sourced example In November 2021 Databricks announced an audited 100 TB TPC-DS result of 32,941,245 QphDS, against 14,861,137 for the previous record holder; the run used Databricks SQL, not a Spark batch job. ## Sources - TPC-DS benchmark page, tpc.org: https://www.tpc.org/tpcds/default5.asp (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/tpc-ds/ --- # TPC-H Glossary, read 10 Nov 2026 **The short answer.** TPC-H is a decision-support benchmark of business-oriented ad hoc queries and concurrent data modifications. Its metric is QphH@Size. Spark accelerator projects use TPC-H-like query sets because they are simple to generate, but the results are not official TPC-H results unless audited. TPC-H queries are fewer and more join-and-aggregate heavy than TPC-DS, so a speedup on one may not carry to the other. When a vendor says a result is derived from TPC-H, treat it as the vendor's own test and look for the data size and scripts. ## A sourced example Google's Lightning Engine launch post reports 3.6x on TPC-H-like workloads at a 10 TB dataset size versus open source Spark on similar infrastructure, and notes the queries are derived from TPC-H. ## Sources - TPC-H benchmark page, tpc.org: https://www.tpc.org/tpch/default5.asp (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/tpc-h/ --- # Scale factor Glossary, read 10 Nov 2026 **The short answer.** Scale factor is the knob that sets how much data a TPC-style benchmark generates. In NVIDIA's NDS tooling, derived from TPC-DS, the scale argument is the volume of data generated in GB. Results at different scale factors measure different things and cannot be averaged. Small scale factors fit in memory and reward low startup cost. Large ones stress shuffle, spill and storage. A result at SF100 says little about a 50 TB table. Always write the scale factor next to a speedup. The Ledger lists it for every entry that publishes it. ## A sourced example NVIDIA's NDS generator takes a scale argument defined as volume of data in GB, and its query-stream script takes a scale to assume a database of that scale factor. ## Sources - NVIDIA NDS benchmark README, GitHub: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/scale-factor/ --- # FPGA Glossary, read 10 Nov 2026 **The short answer.** An FPGA, or field-programmable gate array, is a chip whose logic can be reconfigured after manufacture to run one workload as a hardware pipeline. AWS describes its F2 instances as purpose built to develop and deploy reconfigurable hardware in the cloud, powered by up to 8 AMD Virtex UltraScale+ FPGAs. For Spark, an FPGA option moves sort, join or shuffle work from the CPU onto a pipeline built for it. DualBird says onboarding is an instance type change plus its Spark plugin, with no code changes. DualBird is the FPGA option in this comparison, on AWS F2 only. See [AWS F2 instance](/glossary/aws-f2-instance/). ## A sourced example DualBird says its plugin runs Spark sort, join and shuffle work on FPGAs in AWS F2 instances; its own compaction benchmark reports 12 to 20 times faster single-task execution, a result DualBird ran itself. ## Sources - AWS: Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/fpga/ --- # GPU Glossary, read 10 Nov 2026 **The short answer.** A GPU is a processor with thousands of small cores built for data-parallel work. For Spark, the NVIDIA RAPIDS Accelerator runs supported SQL and DataFrame operators on NVIDIA GPUs and sends the rest back to CPUs. It needs GPU instances. NVIDIA's documentation lists SQL and DataFrame operators as the accelerated path and lists RDDs, GraphX and APIs that take Scala or Java functions as not accelerated. Whether a GPU run is cheaper depends on elapsed time and instance price together. In AWS's August 2026 post, priced at May 2026 On-Demand rates, the 8-node GPU cluster cost $26.35 per hour against $10.43 to $10.99 for the CPU clusters. ## A sourced example AWS documents P5 instances as powered by NVIDIA H100 Tensor Core GPUs, and P5e and P5en instances as powered by H200 GPUs, for deep learning and HPC; Spark clusters often use smaller GPU families. ## Sources - AWS: Amazon EC2 P5 instances: https://aws.amazon.com/ec2/instance-types/p5/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/gpu/ --- # APU (Analytics Processing Unit) Glossary, read 10 Nov 2026 **The short answer.** An APU in this context is Speedata's Analytics Processing Unit, a chip that runs Spark SQL operations in silicon rather than in memory. Speedata says it achieves over a billion rows per second by mapping processing into a hardware pipeline. It needs physical servers or Speedata's hosted service. The C200 card sits in a server's PCIe Gen 5 slot. Speedata's software plugs into Spark's Catalyst optimizer and hands compute-heavy work to the card, with UDFs staying on the CPU. Speedata publishes no price list; none found on 2026-11-10. These are Speedata's claims, not independent tests. ## A sourced example Speedata's blog summary of Microsoft's SIGMOD 2026 paper reports a standalone APU speedup on TPC-H SF100 query 13 of 259.2x versus a single-core SQL Server baseline; that baseline is not a Spark cluster. ## Sources - Speedata: technology page: https://www.speedata.io/technology (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/apu/ --- # AWS F2 instance Glossary, read 10 Nov 2026 **The short answer.** AWS F2 instances are second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB of high-bandwidth memory each, and 3rd-generation AMD EPYC processors. AWS says they deliver up to 60% better price performance than first-generation F1 instances. There are three sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB) and f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). F2 is listed in eight AWS regions. DualBird needs it, so DualBird is available only where F2 is. ## A sourced example The f2.48xlarge size has 8 FPGAs, 192 vCPU, 2,048 GiB of memory, 100 Gbps networking and 60 Gbps of EBS bandwidth, per the AWS instance page. ## Sources - AWS: Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/aws-f2-instance/ --- # EMR release label Glossary, read 10 Nov 2026 **The short answer.** An EMR release label such as emr-7.14.0 names a fixed bundle of open source applications, each at a stated version, that Amazon EMR installs. Upgrading the label is how a team gets a newer Amazon-built Spark runtime on EMR without installing a plugin. Each application carries an Amazon build suffix, such as 3.5.8-amzn-0, showing the version is AWS's build of open source Spark. The release label also fixes Iceberg, Hudi and Hadoop versions. Third-party plugins usually list the Spark version they support; check that against the label's Spark version before testing. ## A sourced example The EMR 7.x application versions table lists emr-7.14.0 with Spark 3.5.8-amzn-0 and Iceberg 1.10.1-amzn-1. ## Sources - AWS: Application versions in Amazon EMR 7.x releases: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/emr-release-label/ --- # EMR on EKS Glossary, read 10 Nov 2026 **The short answer.** Amazon EMR on EKS is a deployment option that runs open source big data frameworks such as Spark on Amazon Elastic Kubernetes Service. EMR builds, configures and manages the containers, and you can run Spark next to other container workloads on one EKS cluster. Pricing differs from EMR on EC2. EMR on EKS charges per vCPU-hour and GB-hour requested, plus EKS and compute costs. Per their own pages, DualBird lists EKS compatibility, DataPelago documents EKS, Flarion lists Spark on Kubernetes, and the NVIDIA GPU plugin ships in a '-spark-rapids' release variant. ## A sourced example AWS and NVIDIA's TPC-DS 3 TB test ran on 8-node EMR on EKS clusters; the GPU nodes were g7.4xlarge, and EMR on EKS added a charge of $2.01 per hour to the cluster. ## Sources - AWS: What is Amazon EMR on EKS: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/emr-on-eks/ --- # Spark Operator Glossary, read 10 Nov 2026 **The short answer.** The Kubernetes operator for Apache Spark, called the Spark Operator, lets you submit and manage Spark applications as Kubernetes custom resources. AWS says EMR releases 6.10.0 and higher support it as a job submission model for EMR on EKS, running the EMR release runtime. With the operator, a SparkApplication YAML file replaces a spark-submit call. Plugins that rely on spark.plugins settings and extra jars work by adding them to the YAML or the image. Flarion lists the Spark Operator or a Helm chart as an install path; other vendors document image-based installs. ## A sourced example The AWS guide shows a SparkApplication run on your own EKS cluster with the EMR 6.10.0-or-later runtime image. ## Sources - AWS: Running Spark jobs with the Spark operator: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/spark-operator/ --- # Iceberg compaction (rewrite_data_files) Glossary, read 10 Nov 2026 **The short answer.** Iceberg compaction rewrites many small data files into fewer large ones. In Spark it is run with the rewrite_data_files procedure, which combines files in parallel to cut metadata overhead and file-open cost. The default strategy is binpack; sort is the alternative, with optional Z-order or Hilbert ordering. Compaction is a Spark job you pay for, repeatedly. Streaming and frequent small writes make it a daily cost, so it is a common target for acceleration. Sort-based compaction shuffles and sorts data, which is the work FPGA and native options say they speed up. ## A sourced example The Iceberg procedure defaults to a 512 MB target (write.target-file-size-bytes, 536870912), rewrites files under 75% of that size and files over 180% of it. ## Sources - Apache Iceberg documentation: Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/iceberg-compaction/ --- # Parquet row group Glossary, read 10 Nov 2026 **The short answer.** A Parquet row group is a horizontal slice of rows inside a Parquet file. A file with N columns is split into M row groups, and each column inside a row group is stored as a column chunk. Readers use footer metadata to skip row groups and read only the column chunks they need. Row group size affects scan speed and parallelism. Small groups add metadata overhead; huge ones need more memory to read and limit how a file can be split. Native readers take advantage of the layout with predicate pushdown and column pruning. ## A sourced example The Parquet format documentation says file metadata is written after the data, to allow single-pass writing, so a reader first reads the footer to locate the column chunks it wants. ## Sources - Apache Parquet documentation: File format: https://parquet.apache.org/docs/file-format/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/parquet-row-group/ --- # Dynamic partition pruning Glossary, read 10 Nov 2026 **The short answer.** Dynamic partition pruning skips partitions of a large table at runtime using the filter results of a joined smaller table. The Spark setting spark.sql.optimizer.dynamicPartitionPruning.enabled is true by default, since Spark 3.0.0. It cuts the data a query reads without any SQL change. Pruning only works on partitioned tables where the join key is the partition column, which is common in star-schema queries. Amazon EMR's release guide documents the feature on its own page, so check whether your baseline had it on. ## A sourced example Per AWS documentation, dynamic partition pruning is enabled by default from Amazon EMR 5.26.0, and can be switched on in 5.24.0 and 5.25.0 with spark.sql.dynamicPartitionPruning.enabled. ## Sources - AWS: Optimize Spark performance on Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/dynamic-partition-pruning/ --- # Cost per job Glossary, read 10 Nov 2026 **The short answer.** Cost per job is the total bill for one run of a Spark job: compute at the instance or per-vCPU rate times elapsed time, plus any platform uplift, storage and license charge. A faster run is cheaper only if the hourly price of the faster setup does not rise by more. Speedup and cost per job are separate numbers. A GPU or FPGA instance can cost more per hour than a CPU instance, so cost per job depends on both. Always ask what the cost figure includes: EBS, S3 requests, the EKS control-plane fee and license fees are often left out. ## A sourced example AWS's EMR on EKS pricing example computes an uplift of $0.4709 for a job using 100 vCPU and 300 GB of memory for 30 minutes, at $0.007084 per vCPU-hour and $0.000777875 per GB-hour. ## Sources - AWS: Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/cost-per-job/ --- # Spark acceleration news, dated and sourced News, read 10 Nov 2026 **The short answer.** Dated items on changes to the twelve options: new releases, a pricing cut, image and runtime support changes, and vendor blog posts. Each item links to the page it comes from and shows the date, so a reader can see when a fact changed and what the source said that day. ## 15 items, newest first - [NVIDIA cuDF plugin for Apache Spark 26.10.0 released, adds Spark 4.2.0 support](/news/nvidia-cudf-plugin-26-10-0/): 5 Nov 2026. NVIDIA published release v26.10.0 of its cuDF plugin for Apache Spark on November 5, 2026. - [Apache DataFusion Comet 1.2.0 released: 399 commits from 40 contributors](/news/comet-1-2-0-released/): 3 Nov 2026. Apache DataFusion Comet 1.2.0 was published on November 3, 2026, four weeks after 1.1.0. - [Apache Spark 4.3.0 released with stateful Real-Time Mode and SCD Type 2](/news/spark-4-3-0-released/): 30 Oct 2026. Apache Spark 4.3.0 was announced on October 30, 2026, the fourth release of the 4.x line. - [Apache Spark 4.2.1, 4.1.4 and 4.0.5 maintenance releases announced](/news/spark-maintenance-releases-oct-2026/): 27 Oct 2026. Apache Spark announced three maintenance releases on October 27, 2026: 4.2.1, 4.1.4 and 4.0.5. - [Amazon EMR on EKS cuts the EMR charge by 30 percent](/news/emr-eks-charge-cut-30-percent/): 23 Oct 2026. AWS announced on October 23, 2026 a 30 percent reduction in the EMR charge added to EKS when running EMR on EKS. - [Apache DataFusion Comet 1.1.0 released: 416 commits from 41 contributors](/news/comet-1-1-0-released/): 6 Oct 2026. Apache DataFusion Comet 1.1.0 was published on October 6, 2026. - [EMR Serverless raises the Spark shuffle limit from 200 GB to 1 TB per job](/news/emr-serverless-terabyte-shuffle/): 1 Oct 2026. AWS announced on October 1, 2026 that Amazon EMR Serverless supports up to 1 TB of shuffle per job, raising the previous 200 GB limit. - [Amazon EMR on EKS adds interactive Spark sessions with Spark Connect](/news/emr-eks-spark-connect-interactive/): 24 Sep 2026. AWS announced on September 24, 2026 that Amazon EMR on EKS supports interactive Apache Spark sessions through Spark Connect. - [Google Lightning Engine reaches Dataproc 3.0 cluster images from 3.0.2](/news/lightning-engine-3-0-images/): 23 Sep 2026. Google's release notes dated September 23, 2026 say Lightning Engine is supported in Dataproc 3.0 cluster images from subminor version 3.0.2 (3.0.2-debian13 and 3.0.2-ubuntu24). - [Amazon EMR introduces Long Term Support, starting with emr-spark-8.1.0 on Spark 4.1](/news/emr-spark-8-1-0-lts/): 22 Sep 2026. Amazon EMR announced Long Term Support releases on September 22, 2026, starting with emr-spark-8.1.0 and Apache Spark 4.1. - [Amazon EMR 7.14 is available with Spark 3.5.8 and Iceberg 1.10.1](/news/emr-7-14-available/): 22 Sep 2026. AWS announced Amazon EMR 7.14 on September 22, 2026. - [Google Lightning Engine runs on Arm with Axion C4A for Dataproc 2.3 clusters](/news/lightning-engine-arm-axion/): 1 Sep 2026. Google's Dataproc release notes of September 1, 2026 say Lightning Engine with Native Query Execution is supported on Arm for image version 2.3 clusters (2.3-ubuntu22-arm). - [DualBird blog Part 2: why data skew costs less on its FPGA architecture](/news/dualbird-skew-part-2/): 1 Sep 2026. DualBird published Part 2 of its Faster Spark series on September 1, 2026, written by its chief architect and co-founder. - [Apache Gluten 1.7.0 released with Spark 4.x test coverage and Delta 4 native write](/news/gluten-1-7-0-released/): 26 Aug 2026. Apache Gluten published release 1.7.0 on August 26, 2026, after three release candidates dated August 10, 17 and 21. - [Flarion blog: the hard part of a Spark upgrade is proving the output did not change](/news/flarion-spark-upgrade-validation/): 18 Aug 2026. Flarion published a blog post on August 18, 2026 arguing that validation, not code rewriting, is the main obstacle to Spark version upgrades. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/ --- # Apache Spark 4.3.0 released with stateful Real-Time Mode and SCD Type 2 News, 30 Oct 2026 **The short answer.** Apache Spark 4.3.0 was announced on October 30, 2026, the fourth release of the 4.x line. The release notes cite over 1,100 Jira tickets and more than 210 contributors, stateful Real-Time Mode at about 100 milliseconds latency, SCD Type 2 in Auto CDC, and a quarterly feature-release cadence. Apache Spark 4.3.0 is the fourth release of the Spark 4.x line, per the project's release notes. The notes say it addressed over 1,100 Jira tickets with contributions from more than 210 individuals. It marks the start of a quarterly feature-release cadence. The headline items are stateful queries in Structured Streaming Real-Time Mode (streaming aggregation, dropDuplicates and transformWithState, with end-to-end latency the notes put on the order of 100 milliseconds), SCD Type 2 in Spark Declarative Pipelines Auto CDC, and opt-in settings that cut fixed overhead on small queries. SQL adds JSON_TABLE, ANSI UNNEST and ASOF JOIN. For a team that accelerates Spark with a plugin or a native engine, the practical check is the plugin's supported-version list. The NVIDIA cuDF plugin page read on 2026-11-10 lists Spark up to 4.2.0, and 4.3.0 is not on that list. None found on 2026-11-10 for the other options. ## Sources - Apache Spark 4.3.0 release notes: https://spark.apache.org/releases/spark-release-4-3-0.html (read 10 Nov 2026) - Apache Spark news: https://spark.apache.org/news/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/spark-4-3-0-released/ --- # Apache Spark 4.2.1, 4.1.4 and 4.0.5 maintenance releases announced News, 27 Oct 2026 **The short answer.** Apache Spark announced three maintenance releases on October 27, 2026: 4.2.1, 4.1.4 and 4.0.5. They cover the 4.2, 4.1 and 4.0 lines, three days before 4.3.0 was announced. The project's news page links each version to its release notes and download page. The Apache Spark project announced Spark 4.2.1, Spark 4.1.4 and Spark 4.0.5 on October 27, 2026. Each announcement points to release notes and a download link on the project's news page. The previous maintenance releases on the same page are Spark 3.5.9 (July 16, 2026), Spark 4.1.3 (July 15, 2026), Spark 4.0.4 (July 15, 2026) and Spark 4.2.0 (July 14, 2026). Three lines moved at once, so a cluster on any 4.x line has a patch to read. Accelerators pin Spark versions closely. The NVIDIA cuDF plugin page read on 2026-11-10 lists Spark 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0, so the three new patch versions are not on its list. Check an option's version list before upgrading Spark under it. ## Sources - Apache Spark news: https://spark.apache.org/news/ (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/spark-maintenance-releases-oct-2026/ --- # Apache Gluten 1.7.0 released with Spark 4.x test coverage and Delta 4 native write News, 26 Aug 2026 **The short answer.** Apache Gluten published release 1.7.0 on August 26, 2026, after three release candidates dated August 10, 17 and 21. The release notes list native Delta 4 write support for Spark 4.0, more Spark 4.0 and 4.1 test suites enabled, and JDK 21 and JDK 25 build support. Apache Gluten tagged v1.7.0 on August 26, 2026. The candidates v1.7.0-rc0, rc1 and rc2 came on August 10, August 17 and August 21. The previous release, v1.6.0, was published on March 10, 2026. The release notes list Delta 4 native write support for Spark 4.0, offload of the Delta OPTIMIZE compaction command, roaring bitmap groundwork for Delta deletion vectors, and many Spark 4.0 and 4.1 test suites enabled for the Velox backend. Build changes include JDK 25 build support and a Java 17 release flag for Spark 4 and later. Gluten is a plugin that runs Spark plans on Velox. A team on Delta tables can read the Velox Delta guide the release adds before testing. Fallback to Spark still applies to what the backend does not support, and the notes list a TimestampNTZ fallback. ## Sources - Apache Gluten v1.7.0 release: https://github.com/apache/gluten/releases/tag/v1.7.0 (read 10 Nov 2026) - Apache Gluten releases: https://github.com/apache/gluten/releases (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/gluten-1-7-0-released/ --- # Apache DataFusion Comet 1.1.0 released: 416 commits from 41 contributors News, 6 Oct 2026 **The short answer.** Apache DataFusion Comet 1.1.0 was published on October 6, 2026. Its changelog states 416 commits from 41 contributors. Listed changes include native Iceberg write fixes, native support for Spark 4 VARIANT columns in Parquet scans, and new native kernels such as make_interval and unbase64. Apache DataFusion Comet 1.1.0 was published on GitHub on October 6, 2026. The changelog says the release consists of 416 commits from 41 contributors. The previous release, 1.0.0, was published on August 7, 2026. Much of the fixed-bugs list concerns native Iceberg writes: partition path escaping, evolved partition specs, cleanup of data files when a write job fails, and fallbacks to Spark for gs:// writes through HadoopFileIO. Enhancements include native scans with unprojected Spark 4 VARIANT columns and exposed native aggregate spill and memory metrics. Comet is a plugin that moves Spark operators to a native DataFusion engine. A team writing Iceberg tables can read the fallback reasons in the changelog, since several entries add a decline or fallback instead of a native path. ## Sources - Apache DataFusion Comet 1.1.0 release: https://github.com/apache/datafusion-comet/releases/tag/1.1.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/comet-1-1-0-released/ --- # Apache DataFusion Comet 1.2.0 released: 399 commits from 40 contributors News, 3 Nov 2026 **The short answer.** Apache DataFusion Comet 1.2.0 was published on November 3, 2026, four weeks after 1.1.0. The changelog states 399 commits from 40 contributors. Entries cover native Iceberg write guards, Spark 4.2 date_trunc matching, memory-limit sizing for YARN and Kubernetes containers, and ANSI error handling. Apache DataFusion Comet 1.2.0 was published on November 3, 2026. The changelog says it consists of 399 commits from 40 contributors. Release 1.1.0 was published on October 6, 2026. Listed fixes include deriving the native memory limit the way YARN and Kubernetes size executor containers, returning a native plan's memory before its Spark task ends, and falling back from the native Iceberg scan when a nested field was added or renamed. Others match Spark 4.2 behavior for date_trunc overflow and preserve Spark 4.2 Iceberg write transactions. For a team on EKS or YARN, the container memory entry is the one to read against its executor settings. Nothing in the changelog states a speedup figure. ## Sources - Apache DataFusion Comet 1.2.0 release: https://github.com/apache/datafusion-comet/releases/tag/1.2.0 (read 10 Nov 2026) - Apache DataFusion Comet releases: https://github.com/apache/datafusion-comet/releases (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/comet-1-2-0-released/ --- # NVIDIA cuDF plugin for Apache Spark 26.10.0 released, adds Spark 4.2.0 support News, 5 Nov 2026 **The short answer.** NVIDIA published release v26.10.0 of its cuDF plugin for Apache Spark on November 5, 2026. The download page, which renames the RAPIDS Accelerator, lists Spark 3.3.0 through 4.2.0, Databricks 14.3 and 17.3 ML LTS GPU runtimes, and CUDA 12 (driver R525+) and CUDA 13 (R580+) jar variants. NVIDIA published v26.10.0 on its cudf-spark GitHub releases page on November 5, 2026. The earlier releases in the window are v26.08.0 (August 20, 2026) and v26.08.1 (August 28, 2026). The release body points to the download page for details. That page says the plugin was formerly the RAPIDS Accelerator for Apache Spark, that the RAPIDS name will be sunset over time, and that artifact names remain the same for now. It lists supported Spark versions up to 4.2.0, tested GPUs from V100 to RTX PRO 6000, and Databricks runtimes 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). A team that needs a GPU should check the runtime and driver rows on the page against its cluster images. EMR support is documented in the project's FAQ, per the page. ## Sources - NVIDIA cudf-spark v26.10.0 release: https://github.com/NVIDIA/cudf-spark/releases/tag/v26.10.0 (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/nvidia-cudf-plugin-26-10-0/ --- # Amazon EMR introduces Long Term Support, starting with emr-spark-8.1.0 on Spark 4.1 News, 22 Sep 2026 **The short answer.** Amazon EMR announced Long Term Support releases on September 22, 2026, starting with emr-spark-8.1.0 and Apache Spark 4.1. AWS says LTS versions get 36 months of support at no additional cost, plus Apache Iceberg v3 support, in every Region where EMR is available across EC2, EKS and Serverless. AWS announced on September 22, 2026 that Amazon EMR now has Long Term Support releases, with emr-spark-8.1.0 on Apache Spark 4.1 the first. Per AWS, designated versions of the AWS runtime for Apache Spark receive 36 months of support, with fixes for critical and high severity security, bug and data-corruption issues, subject to availability, at no additional cost. AWS lists full support for Apache Iceberg v3, Spark SQL references to catalogs by name with automatic detection of Iceberg, Delta Lake and Hudi tables, wider fine-grained access control for Iceberg operations and Delta Lake VACUUM, and Spark Connect endpoints on EMR on EKS. It is offered on EMR on EC2, EMR on EKS and EMR Serverless. A team that upgrades Spark rarely can read the 36-month window against its own upgrade calendar. AWS also points to the Apache Spark Upgrade Agent for Amazon EMR for moving existing applications. ## Sources - AWS What's New: EMR Long Term Support with Apache Spark 4.1: https://aws.amazon.com/about-aws/whats-new/2026/09/amazon-emr-long-term-support-spark-4-1/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/emr-spark-8-1-0-lts/ --- # Amazon EMR 7.14 is available with Spark 3.5.8 and Iceberg 1.10.1 News, 22 Sep 2026 **The short answer.** AWS announced Amazon EMR 7.14 on September 22, 2026. It upgrades Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1, adds Spark Connect and IPv6 support on EMR on EKS, and raises the EMR Serverless storage limit for Spark jobs from 200 GiB to 1 TiB for shuffle data. AWS posted the Amazon EMR 7.14 announcement on September 22, 2026. The release upgrades Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1, and is available in all Regions where EMR is available. The release guide's lifecycle table lists an initial release date of September 8, 2026 and standard support until September 8, 2028. AWS says Iceberg materialized views now refresh faster on tables with updates and deletes, using change data capture to read only affected data files. On EMR on EKS, clusters get Spark Connect endpoints with token-based authentication and can run on IPv6 EKS clusters. On EMR Serverless, the storage limit for Spark jobs grows from 200 GiB to 1 TiB. The 7.x line stays on Scala 2.12.18 per the application-versions table, while the new emr-spark-8.1.0 line moves to Spark 4.1. Both lines appear in the release guide for a team planning an upgrade. ## Sources - AWS What's New: Amazon EMR 7.14: https://aws.amazon.com/about-aws/whats-new/2026/09/amazon-emr-7-14-available/ (read 10 Nov 2026) - Amazon EMR 7.14.0 release guide: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-7140-release.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/emr-7-14-available/ --- # Amazon EMR on EKS cuts the EMR charge by 30 percent News, 23 Oct 2026 **The short answer.** AWS announced on October 23, 2026 a 30 percent reduction in the EMR charge added to EKS when running EMR on EKS. The cut applies to both the per-vCPU-hour and per-GB-hour charges, while underlying Amazon EKS charges stay unchanged. AWS says it applies in all Regions where EMR on EKS is available. AWS announced a 30 percent reduction in the EMR charge that is added to the EKS cost when running Spark on EMR on EKS. The post is dated October 23, 2026. It applies to both the per-vCPU-hour charge and the per-GB-hour charge. The underlying Amazon EKS charges remain unchanged, and the new price applies automatically to existing EMR on EKS workloads with no action required, per AWS. It covers all Regions where EMR on EKS is available. The post gives no new dollar rates. A team that tracks the bill by line item should read the Amazon EMR pricing page for the new per-vCPU-hour and per-GB-hour figures before updating its cost model, since the EC2 and EKS lines are not part of this change. ## Sources - AWS What's New: EMR on EKS reduces EMR charge by 30 percent: https://aws.amazon.com/about-aws/whats-new/2026/10/emr-eks-reduces-emr-charge-30-percent/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/emr-eks-charge-cut-30-percent/ --- # EMR Serverless raises the Spark shuffle limit from 200 GB to 1 TB per job News, 1 Oct 2026 **The short answer.** AWS announced on October 1, 2026 that Amazon EMR Serverless supports up to 1 TB of shuffle per job, raising the previous 200 GB limit. It needs emr-7.14 or emr-spark-8.1 and later and is offered in 18 AWS Regions, with spill support for memory-heavy operations. AWS posted on October 1, 2026 that serverless storage on Amazon EMR Serverless now supports up to 1 TB of shuffle operations, raising the previous 200 GB per-job limit. AWS names large table joins across multi-terabyte datasets and aggregations on high-cardinality data as the target workloads. The feature needs emr-7.14, emr-spark-8.1 or later and is available in 18 AWS Regions where EMR Serverless is available. AWS adds spill support so memory-intensive operations can offload data to disk. A team whose nightly jobs fail on shuffle storage can compare this limit with the shuffle volume in its Spark UI. The post gives no timing or cost figures for the larger limit. ## Sources - AWS What's New: EMR Serverless terabyte-scale shuffle: https://aws.amazon.com/about-aws/whats-new/2026/10/emr-serverless-terabyte-scale-shuffle/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/emr-serverless-terabyte-shuffle/ --- # Amazon EMR on EKS adds interactive Spark sessions with Spark Connect News, 24 Sep 2026 **The short answer.** AWS announced on September 24, 2026 that Amazon EMR on EKS supports interactive Apache Spark sessions through Spark Connect. It runs with EMR 7.14 (Spark 3.5) and emr-spark-8.1.0 (Spark 4.1) in all AWS Commercial Regions, from SageMaker Unified Studio notebooks or IDEs such as Jupyter and Visual Studio Code. AWS posted on September 24, 2026 that Amazon EMR on EKS now supports interactive Spark sessions with Spark Connect. Data engineers can develop and debug from managed notebooks in Amazon SageMaker Unified Studio and from IDEs such as Jupyter and Visual Studio Code, with Spark running on the EKS clusters they already operate. Each session runs as pods on a virtual cluster, secured with an IAM execution role and tagged by project and user. It is available with EMR release 7.14 (Apache Spark 3.5) and emr-spark-8.1.0 (Apache Spark 4.1) in all AWS Commercial Regions. A team that uses a Spark plugin on EKS can check whether the plugin's session mode works over Spark Connect. Nothing in the post addresses plugins. ## Sources - AWS What's New: Spark Connect on Amazon EMR on EKS: https://aws.amazon.com/about-aws/whats-new/2026/09/emr-eks-spark-connect-interactive/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/emr-eks-spark-connect-interactive/ --- # Google Lightning Engine runs on Arm with Axion C4A for Dataproc 2.3 clusters News, 1 Sep 2026 **The short answer.** Google's Dataproc release notes of September 1, 2026 say Lightning Engine with Native Query Execution is supported on Arm for image version 2.3 clusters (2.3-ubuntu22-arm). It targets vectorized execution on Google Axion C4A instances for Spark SQL, Cloud Storage Parquet, Apache Iceberg and Delta Lake workloads. The Google Cloud release notes for Managed Service for Apache Spark (formerly Dataproc on Compute Engine) list, under September 1, 2026, that Lightning Engine with Native Query Execution (NQE) is now supported on Arm. The image is 2.3-ubuntu22-arm. Google describes it as accelerated vectorized query execution on Google Axion (C4A) instances for Apache Spark SQL, Cloud Storage Parquet, Apache Iceberg and Delta Lake workloads. The note gives no speedup figure for the Arm image. A team on Google Cloud can read the image version and instance family against its cluster templates. ## Sources - Google Cloud Dataproc release notes: https://cloud.google.com/dataproc/docs/release-notes (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/lightning-engine-arm-axion/ --- # Google Lightning Engine reaches Dataproc 3.0 cluster images from 3.0.2 News, 23 Sep 2026 **The short answer.** Google's release notes dated September 23, 2026 say Lightning Engine is supported in Dataproc 3.0 cluster images from subminor version 3.0.2 (3.0.2-debian13 and 3.0.2-ubuntu24). The same entry sets Iceberg 1.10 and Parquet footer caching as defaults for Lightning Engine in the 2.3 serverless runtime. Under September 23, 2026, the Google Cloud release notes announce that Lightning Engine is supported in Managed Service for Apache Spark cluster image 3.0 starting with subminor 3.0.2, in Debian 13 and Ubuntu 24 variants. The new serverless runtime versions are 1.2.89, 2.2.89 and 2.3.42. In the 2.3 runtime, Serverless batches with Lightning Engine use Apache Iceberg 1.10 by default, and Parquet footer caching is enabled by default for Lightning Engine (Velox). Google says that if executor out-of-memory or task failures spike on tiny files, setting spark.gluten.sql.columnar.backend.velox.cacheParquetFooters=false mitigates it. The notes give no benchmark for the 3.0 images. An older entry on the same page, dated October 22, 2025, quotes Google's own figure of up to 4.3x faster than open-source Spark on TPC-H-like benchmarks, and says it requires the Premium tier on Serverless. ## Sources - Google Cloud Dataproc release notes: https://cloud.google.com/dataproc/docs/release-notes (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/lightning-engine-3-0-images/ --- # DualBird blog Part 2: why data skew costs less on its FPGA architecture News, 1 Sep 2026 **The short answer.** DualBird published Part 2 of its Faster Spark series on September 1, 2026, written by its chief architect and co-founder. The post says Spark gives each running task about 1/N of the shared execution memory, and argues its FPGA architecture reduces the impact of hot-key skew without repartitioning or salting. DualBird published Faster Spark Was the Goal. A Simpler Spark Came with It: Part 2 on September 1, 2026. The author is listed as Ehud Eliaz, Chief Architect and Co-Founder. The post is about hot-key skew, not high cardinality. It states that in standard Spark each running task gets roughly 1/N of the executor's shared execution memory regardless of its data, and that a hot key concentrates in one shuffle-read task, so a stage waits for its slowest task. DualBird says its dedicated hardware (FPGA) architecture makes the same skewed data much less of a problem. This is DualBird's own argument. The post as read on 2026-11-10 gives no benchmark figure in the portion read. A team with skewed joins can compare it with Spark's adaptive query execution skew join handling, which the post says covers joins only. ## Sources - DualBird blog: Faster Spark Was the Goal, Part 2: https://dualbird.io/resources/blog/faster-spark-part2 (read 10 Nov 2026) - DualBird resources: https://dualbird.io/resources (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/dualbird-skew-part-2/ --- # Flarion blog: the hard part of a Spark upgrade is proving the output did not change News, 18 Aug 2026 **The short answer.** Flarion published a blog post on August 18, 2026 arguing that validation, not code rewriting, is the main obstacle to Spark version upgrades. It cites Slack's Spark 2 to 3 move across 60+ EMR clusters and 40+ teams, and Uber's 2.4 to 3.3 upgrade with over two million applications daily. Flarion's blog post of August 18, 2026, by Ran Reichman, responds to the AWS Spark Upgrade Agent, which it says rewrites Spark jobs for Spark 4.0. Flarion's argument is that the code rewrite is not the main impediment to upgrading enterprise Spark workloads. The post says Slack's Spark 2 to 3 migration took about a year across 60+ EMR clusters and 40+ teams, with exact-match checks for billing pipelines. It says Uber moved from Spark 2.4 to 3.3 with over two million Spark applications running daily, using a shadow-testing framework to compare outputs before marking a job migrated. These are Flarion's reading of those companies' write-ups. Flarion sells a Spark plugin, so the post sits next to its product pitch. A team weighing an upgrade or a plugin can use the same question for both: how would the output be compared before and after. ## Sources - Flarion blog: The Hard Part of Upgrading Spark Versions Isn't the Code: https://www.flarion.io/blog/the-hard-part-of-upgrading-spark-versions-isnt-the-code (read 10 Nov 2026) - Flarion blog: https://www.flarion.io/blog (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/news/flarion-spark-upgrade-validation/ --- # Questions about speeding up Spark, answered from published facts FAQ, read 10 Nov 2026 **The short answer.** Thirty questions a data engineer asks about speeding up Spark without rewriting jobs, answered with sourced facts: what no code change means, which clouds each option supports, how benchmarks are run, what fallback costs, where prices are published and what none found means. ## 30 questions and answers **What does each kind of Spark accelerator actually change?** Plugins and native engines (Apache Auron, Apache Gluten, DataFusion Comet, Flarion) swap Spark's JVM execution for C++ or Rust code behind a spark.plugins setting. Platform runtimes (Databricks Photon, the Amazon EMR runtime, Google's Lightning Engine, Fabric's native engine) are built into one cloud service. Hardware options (NVIDIA RAPIDS, DualBird, Speedata) add GPUs, FPGAs or a card. See the [Change Budget](/change-budget/) for code, cluster, cloud and vendor changes. **Which options run on Amazon EMR or EMR on EKS?** Per each vendor's own pages: DataPelago documents Spark on EKS, DualBird says it is compatible with Amazon EMR and EKS, Flarion lists an EMR bootstrap action, and the NVIDIA RAPIDS Accelerator is supported on EMR on EKS from release 6.9.0. The Amazon EMR runtime is AWS's own. Auron, Comet and Gluten document no EMR support; none found on 2026-11-10. **Which options are open source?** Apache Auron, Apache Gluten and DataFusion Comet are Apache-2.0 projects. The NVIDIA RAPIDS Accelerator source is Apache-2.0 and its documentation is CC-BY-4.0. Databricks Photon, the Amazon EMR runtime, DataPelago, Flarion and Speedata are proprietary. For DualBird, no public repository was found on 2026-11-10. Google's Lightning Engine and Fabric's engine are cloud services, though Fabric's builds on Velox and Gluten. **Which options need GPUs or special instances?** The NVIDIA RAPIDS Accelerator needs NVIDIA GPUs (Volta through Blackwell generations). DualBird needs AWS F2 FPGA instances. Speedata needs its C200 APU card in a physical server or its hosted service. Google's native query execution can run on Axion ARM (C4A) machines but also on x86. Comet, Flarion and Gluten run on standard CPU instances. DataPelago's DPA-S docs state no GPU requirement, though its engine targets CPU, GPU and FPGA. **How does operator fallback work?** When a plugin cannot run an operator or expression natively, Spark runs that part on the JVM and the plugin runs the rest. The job returns a result, but the fallback stage gets no speedup, and data may be converted between columns and rows. NVIDIA says unsupported operators and UDFs fall back to the CPU. Gluten documents fallback through columnar-to-row conversions. **How should I read a Spark acceleration benchmark?** Check who ran it, the baseline Spark version and settings, the data size or scale factor, the number of queries and whether scripts are published. A vendor's own run is the vendor's claim. NVIDIA, for example, notes its NDS comparison used different hardware and software stacks. The [Ledger](/ledger/) lists these fields for every published entry we found. **Why are most Spark benchmark results not official TPC results?** The TPC audits results and publishes a report for each. Most Spark accelerator tests are runs by a vendor or project using TPC-derived queries, written TPC-DS-like, TPC-H-like or NDS. NVIDIA's own disclaimer on NDS is one example of that labeling. Databricks's audited 100 TB TPC-DS result of November 2021 ran on Databricks SQL, not a Spark batch job. **What is DualBird and what does it require?** DualBird says its Spark plugin runs sort, join and shuffle work on FPGAs in AWS F2 instances. DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. **What does DualBird claim about speed and cost?** DualBird says 10x-30x faster data processing with 50%-90% lower costs and no changes required. The meta description on the same page says 10x-100x. For Iceberg compaction on 100 GB DualBird reports roughly 12-20x faster single-task execution and about $2 per terabyte against about $6 per terabyte, rising past $13, for CPU clusters. DualBird ran these tests itself. **What does DualBird say a team has to change?** DualBird says onboarding is two steps: change the EC2 instance type to F2 and add its Spark plugin. It also says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. DualBird publishes no Spark version list; none found on 2026-11-10. **Which AWS F2 sizes exist?** AWS lists three F2 sizes: f2.6xlarge with 1 FPGA, 24 vCPU and 256 GiB of memory; f2.12xlarge with 2 FPGAs, 48 vCPU and 512 GiB; and f2.48xlarge with 8 FPGAs, 192 vCPU and 2,048 GiB. Each FPGA is an AMD Virtex UltraScale+ HBM VU47P with 16 GiB of high-bandwidth memory. **What does AWS say its own EMR Spark runtime does?** AWS says the performance-optimized EMR runtime executes up to 4.5x faster than open source equivalents, and writes to Iceberg up to 2.7x faster. The page gives no TPC-DS scale or cluster for those figures. AWS publishes a benchmark repository, aws-samples/emr-spark-benchmark, with a 3 TB TPC-DS dataset and the commands to run it yourself. **What did AWS and NVIDIA publish for GPU Spark on EMR on EKS?** AWS reports TPC-DS at 3 TB on 8-node EMR on EKS clusters: 281 seconds on g7.4xlarge GPU instances against 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. The test used Spark 3.5.6, the median of 5 iterations, in us-east-1. NVIDIA's own overview adds separate NDS runs on different hardware. **What did the GPU test cost, from published prices?** AWS reports $2.06 per benchmark run on the GPU cluster against $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The GPU cluster cost $26.35 per hour (8 x $3.042 for EC2 plus $2.01 for EMR on EKS); the CPU clusters cost $10.43 to $10.99 per hour. AWS counts neither the EKS control-plane fee, EBS, S3 nor the driver pod. **How does AWS charge for EMR on EKS?** AWS charges EMR on EKS by the vCPU and memory a job requests, from image download until the pod ends, rounded up to the nearest second, added to EKS and compute costs. Its pricing example: 100 vCPU and 300 GB for 30 minutes at $0.007084 per vCPU-hour and $0.000777875 per GB-hour is a $0.4709 EMR uplift. Each EKS cluster also costs $0.10 per hour. **What does Google's Lightning Engine cost?** Google's pricing page gives example us-central1 rates for serverless Managed Service for Apache Spark: $0.06 per DCU-hour on the standard tier and $0.089 per DCU-hour on the premium tier that includes Lightning Engine. Billing is per second with a one-minute minimum. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. It runs on Google Cloud only. **What does Flarion charge?** Flarion publishes usage-based list prices: $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP, $0.012 per CPU hour for fully managed Spark such as EMR, $0.095 per DBU on Databricks, and $0.75 per CPU per month on premises. Billing is on accelerated Spark cores, driver plus executors. That is the software fee only; cluster costs are separate. **Are prices published for DualBird, DataPelago, Speedata and Photon?** DualBird: none found on 2026-11-10. Speedata: no price list; the pages offer a server purchase or partner routes. DataPelago: pay-as-you-go billing through the AWS and GCP marketplaces, with no unit price on the pages read. Databricks Photon: instance types use DBUs at a different rate than non-Photon, so the DBU multiplier on Databricks's pricing page sets the cost. **Which Spark versions do the open source plugins support?** From each project's own documents: Apache Auron lists Spark 3.0 to 3.5 and 4.0 to 4.2 in its build script; Apache Gluten officially supports 3.4.4, 3.5.5, 4.0.2 and 4.1.1; DataFusion Comet supports 3.5.9, 4.0.4, 4.1.3 and 4.2.0; the NVIDIA RAPIDS Accelerator v26.10.0 supports 3.3.x through 4.2.0. Check your EMR release label's Spark build before testing. **Which options document Apache Iceberg support?** DataFusion Comet has a native Iceberg reader tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11 and native writes that are off by default. Apache Auron added native scans for Iceberg, Hudi and Paimon in 8.0.0-incubating. Gluten's Velox backend reads Iceberg tables only. Databricks says Photon accelerates Iceberg writes. DualBird mentions Iceberg in its own compaction benchmark and in the description of a December 2025 demo post. **Do any options work on Databricks or Microsoft Fabric?** Databricks Photon is built into Databricks compute. Microsoft's Fabric native execution engine runs only inside Fabric, on Runtime 1.3 (Spark 3.5) and Runtime 2.0 (Spark 4.1). Flarion lists Databricks init scripts, and the NVIDIA RAPIDS Accelerator lists Databricks runtimes 14.3 ML LTS and 17.3 ML LTS. DualBird is AWS F2 only. **Do I need to change memory settings after adding a plugin?** Usually yes. Gluten's quick start sets spark.memory.offHeap.size=20g. Comet's benchmark used 32G off-heap with 64G executors. Auron instead disables Spark off-heap memory and raises executor overhead, 4096 MB in its quick start. Spark's off-heap size defaults to 0 and must be positive when off-heap is enabled. Treat memory sizing as part of the trial. **Do UDFs and RDD code get accelerated?** Mostly no. NVIDIA says UDFs fall back to the CPU and RDDs are not accelerated. Databricks's Photon page lists UDFs, RDD APIs and Dataset APIs as unsupported. Google does not recommend its native query execution for RDD-heavy jobs or UDFs. DataPelago advises avoiding heavy RDD and UDF use. Flarion says UDFs and UDAFs keep running on Spark as before. **Are results identical to plain Spark?** Not always. NVIDIA aims for bit-for-bit identical results and documents operator compatibility. Comet leaves some natively different expressions off unless allowIncompatible is set. Gluten's regexp functions use RE2, which has no lookaround, and ANSI mode always falls back. DataPelago warns of possible incorrect results in case-sensitive mode. Compare output on your own data first. **Can I test an option before changing production?** Several publish tools. NVIDIA has a Qualification tool and an AutoTuner, Google has a qualification tool for native query execution, Speedata offers a free Workload Analyzer that reads Spark logs, and Flarion offers a free assessment that simulates each job from logs. Flarion also says one config change switches it off to compare. **What does Speedata's APU require?** Speedata says its Dash software plugs into Spark's Catalyst optimizer and offloads compute-heavy work to its APU, with UDFs delegated to the CPU. The APU is a PCIe Gen 5 card (C200), so it needs physical servers: a 2U server from Speedata, servers from Dell or HPE, or Speedata's hosted service for AWS. Speedata's pages state no EMR, EKS or Databricks support; none found on 2026-11-10. **What does the Spark Accelerator Ledger list?** The [Ledger](/ledger/) lists each published benchmark for each option: the workload, data size, hardware, result, cost where published, who ran it and whether they are the vendor, and whether the method is public. It lists only numbers a named party published, with the link and date. No number is averaged or estimated. Read the [Ledger method](/ledger/method/) for the entry rules. **What does the Change Budget show?** The [Change Budget](/change-budget/) shows, for each option, what a team has to change: code, cluster, cloud or vendor. Many options need no job code changes but do change the cluster, such as the instance type, memory settings or a plugin jar. Each cell links to the vendor's or project's own page. **Does Adaptive Query Execution already do this work?** No, AQE and accelerators work at different layers. AQE re-plans a query at runtime from statistics: it coalesces small shuffle partitions, splits skewed ones and can turn a sort-merge join into a broadcast join. It is on by default since Spark 3.2.0. Accelerators change how the chosen plan executes. EMR documents AQE and dynamic partition pruning as part of its own runtime. **What does a Spark plugin install actually involve?** Usually a jar on the classpath, the spark.plugins setting, memory changes and, for some, a custom shuffle manager. Gluten uses org.apache.gluten.GlutenPlugin and ColumnarShuffleManager. Speedata uses spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. Spark documents the plugin interface as a DeveloperApi, so check the supported Spark versions first. ## Sources - Apache Gluten home page: https://gluten.apache.org/ (read 10 Nov 2026) - AWS Big Data Blog: GPU-accelerated Apache Spark with Amazon EMR: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - NVIDIA cudf-spark repository LICENSE: https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE (read 10 Nov 2026) - NVIDIA RAPIDS Accelerator for Apache Spark: download and requirements: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - NVIDIA Spark RAPIDS user guide: overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - NVIDIA NDS benchmark README: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - DualBird blog: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird blog: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - AWS: Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - AWS: Amazon EMR features for Apache Spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - AWS: Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Google Cloud: Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - Flarion: pricing: https://www.flarion.io/pricing (read 10 Nov 2026) - Databricks: pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Apache DataFusion Comet: installation guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache DataFusion Comet: Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Microsoft Learn: Native execution engine for Fabric Spark: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Apache Spark configuration reference: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026) - NVIDIA Spark RAPIDS user guide: FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - Apache Gluten: Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Speedata: Workload Analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Speedata: product page: https://www.speedata.io/product (read 10 Nov 2026) - Apache Spark documentation: SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026) - Apache Spark 4.3.0 API: SparkPlugin: https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/plugin/SparkPlugin.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/faq/ --- # Five situations, what to check in each Situations, read 10 Nov 2026 **The short answer.** Five buyer situations: a Spark bill that doubled on EMR or EKS, nightly pipelines that miss the morning, AI data preparation that waits on Spark, a move off a managed platform, and daily Iceberg compaction. Each page lists what to check and which published facts answer it, by option. ## Five situations - [Spark bill doubled in a year on EMR or EKS](/situations/bill-doubled/): Five checks for a data platform team on EMR or EKS whose Spark bill doubled, each answered with published, dated facts from the twelve options. - [Nightly pipelines that miss the morning](/situations/late-pipelines/): Five checks for a team whose nightly Spark pipelines overrun, with the runtimes and fallback limits each vendor has published. - [Data preparation for AI and feature pipelines](/situations/ai-data-prep/): Five checks for a team whose AI and feature pipelines wait on Spark data preparation, covering workload claims, UDFs, file formats, hardware and cloud. - [Leaving a managed platform for an open stack](/situations/leaving-managed/): Five checks for a team moving Spark from a managed platform to an open stack: what is lost, what runs outside, what you operate, license terms and versions. - [Paying every day to compact Iceberg tables](/situations/iceberg-compaction/): Five checks for a team that pays for Iceberg compaction every day: published compaction results, Iceberg support, cost per terabyte, test design and location. Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/ --- # What to check when the Spark bill on EMR or EKS has doubled Situation, read 10 Nov 2026 **The short answer.** A team on EMR or EKS whose Spark bill doubled can add an accelerator, but each option moves cost in a different place: instance type, license fee or platform charge. This page lists five checks and the published figures that answer them, from AWS's $2.06 per run on a GPU cluster to Flarion's $0.009 per CPU hour. ## 5 things to check, with the published facts that answer each ### Which options publish a price or a cost per run, and what does it leave out? A speed claim does not become a bill until the price of the plugin, the instances and the platform are all on the page. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS says the EMR price is added to the EC2 and EBS price, and EMR on EKS is charged per vCPU-hour and GB-hour requested plus EKS and EC2 or Fargate cost. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | A cost-per-job result for Auron: none found on 2026-11-10. The project's only resource claim is about 50% fewer cluster resources. | [Apache Auron homepage](https://auron.apache.org/), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | A cost-per-job result for Comet: none found on 2026-11-10. | [Comet benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | A cost-per-job result for Gluten: none found on 2026-11-10. The software is free under Apache-2.0. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago describes pay-as-you-go billing through the AWS and GCP marketplaces; no unit price is published on the pages read. | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird publishes percentages (50% to 90% lower costs, 55% to 85% lower EC2 cost) and one figure of about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. No price for the plugin is published, so it is not stated whether the F2 instance price or a license fee is inside the savings. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP and $0.012 for fully managed Spark such as EMR, billed on accelerated Spark cores. | [Flarion pricing page](https://www.flarion.io/pricing), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google lists $0.06 per DCU-hour standard and $0.089 premium (Lightning Engine) for serverless in us-central1; clusters add a management fee, a Lightning Engine add-on and Compute Engine costs. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | AWS reports $2.06 per TPC-DS 3 TB run on the GPU cluster against $2.93 to $3.18 on CPU clusters, using On-Demand us-east-1 prices as of May 2026. The cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata publishes no price list; its claims are up to 90% lower TCO and 37 servers replaced by 3, and hardware must be bought or rented. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Does it run on the EMR or EKS setup you already have? An option that needs a different runtime or a different Spark build changes what your jobs run on, not just what they run with. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS lists the runtime on EMR on EC2, EMR on EKS and EMR Serverless, and calls it 100% API compatible with open source Spark. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | Auron's docs describe a jar on the Spark classpath. Documented support or a tested setup for Amazon EMR, EKS or Databricks: none found on 2026-11-10. | [Apache Auron README](https://raw.githubusercontent.com/apache/auron/master/README.md), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's TPC-DS run used an EKS cluster on r6i instances. The docs warn it may not fully work with cloud-provider Spark forks, so EMR is not documented as supported. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten's docs do not mention EMR, EKS or Databricks support; the stated official Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1. | [Gluten Velox backend guide](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago ships DPA-S on AWS as a Docker container in ECR for OSS Spark on EKS (Spark 3.5.1, Java 11, Scala 2.12). EMR and Databricks are not mentioned on the AWS docs page. | [DataPelago AWS deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. Spark version numbers and table format support lists: none found on 2026-11-10. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists Amazon EMR (bootstrap action) and Spark on Kubernetes (Helm chart or Spark operator) among its platforms, and publishes no supported Spark version list. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | Amazon EMR on EKS supports the plugin natively from release 6.9.0, and the '-spark-rapids' release variant ships it pre-installed. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata says Dash accelerates Spark 3.x on Kubernetes, YARN and standalone managers. EMR and EKS support is not stated on the pages read. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What instance or hardware change comes with it? The bill moves with the instance family. A change to GPU or FPGA instances is a capacity and availability question as well as a price one. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | Using the EMR runtime means running on EMR (on EC2, EKS or Serverless); AWS states no instance family change. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's Maven Central jars bundle native libraries for Linux on amd64 and arm64. No GPU or special hardware is required. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird needs AWS EC2 F2 instances: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs) and f2.48xlarge (8 FPGAs, 192 vCPU). F2 is listed in eight regions, not including us-east-2 or eu-west-1. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion says it runs CPU-native on the fleet already in use, and contrasts that with GPU accelerators. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | The AWS test used 8 x g7.4xlarge at $26.35 per hour for the cluster, against $10.43 to $10.99 per hour for the CPU clusters. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata's C200 is a PCIe Gen 5 card in a physical server. The AWS option runs the work in a Speedata-hosted environment that syncs data from S3. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Is the baseline in the benchmark the Spark you run? Speedups are ratios. A slow or unusual baseline inflates the ratio without changing your bill. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS says up to 4.5x faster than open-source equivalents; the page does not state the TPC-DS scale or cluster. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | The Comet project's own page shows the baseline matters: Spark 4.2.0 took 994 s and Spark 4.1.3 took 1,336 s on the same cluster, so Comet 1.2.0 is published as 2.0x and 2.51x against them (488 s and 532 s). The project attributes the gap mostly to the default S3A input stream. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's baseline wording varies: the compaction post compares with vanilla Spark and an unnamed C++ accelerated Spark, while an April 2026 post says a high-end C++ columnar engine on a single CPU core. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion ran Spark 3.5.5 as its baseline on one i4i.8xlarge: 135.6 min at 1 TB against 38.3 min for Flarion. | [Flarion benchmarks page](https://www.flarion.io/benchmarks), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google compares with open source Spark on similar infrastructure (3.6x on TPC-H-like queries at 10 TB) and publishes no scripts, cluster spec or raw timings. | [Google Lightning Engine launch blog](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | AWS compares GPU instances with three CPU instance families (m9gd, m8id, m8a) at 3 TB; NVIDIA's own page notes its EC2 comparison uses different hardware and software stacks. | [NVIDIA Spark plugin overview](https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Can you test one job and switch it off? A trial on a single nightly job produces a cost figure for your own workload, which no published benchmark does. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron describes fine-grained fallback for unsupported operators, and its Spark UI has an Auron tab that records per-operator fallback reasons. | [Apache Auron homepage](https://auron.apache.org/), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's plan explain output marks fallback reasons with [COMET: ...]. | [Comet compatibility guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion says you can turn it off with one config change to compare, and offers a free assessment that simulates each job from logs. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents a qualification tool for checking jobs before enabling the engine at cluster creation. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | AWS recommends a pre-migration assessment with the NVIDIA qualification tool and spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on the CPU. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata offers a free Workload Analyzer: upload Spark logs, run a CLI locally or compare against TPC-DS. | [Speedata Workload Analyzer page](https://www.speedata.io/workload-analyzer), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Apache Auron homepage: https://auron.apache.org/ (read 10 Nov 2026) - Comet benchmarking guide: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - DataPelago Accelerator for Spark page: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Flarion pricing page: https://www.flarion.io/pricing (read 10 Nov 2026) - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Apache Auron README: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Comet installation guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Gluten Velox backend guide: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) - DataPelago AWS deployment docs: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - Speedata technology page: https://www.speedata.io/technology (read 10 Nov 2026) - AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Google Lightning Engine launch blog: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - NVIDIA Spark plugin overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026) - Comet compatibility guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Speedata Workload Analyzer page: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/bill-doubled/ --- # What to check when nightly Spark pipelines no longer finish before morning Situation, read 10 Nov 2026 **The short answer.** A team whose nightly pipelines overrun needs a shorter wall-clock time, not a smaller bill. The published runtimes range from Comet's 994 s to 488 s and NVIDIA's 1,010 s to 281 s on TPC-DS to Flarion's 135.6 min to 38.3 min at 1 TB, each on a different cluster. ## 5 things to check, with the published facts that answer each ### Which runtimes have been published, on what workload and size? Compare seconds and minutes only within one test. Each test below used its own cluster and baseline. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS says up to 4.5x faster than open-source equivalents, with no workload or scale stated. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | The Auron project says about 2x faster than Spark 3.5 on TPC-DS 1 TB (Spark 3.5.6 against Auron 6.0.0-preview on YARN); per-query numbers are a chart image only. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | TPC-DS derived, scale factor 1000, EKS on r6i: Spark 4.2.0 994 s, Comet 1.1.0 601 s, Comet 1.2.0 488 s. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | The Gluten homepage reports 3.34x on TPC-H and 3.02x on TPC-DS at 3 TB with Spark 3.3.1 on Intel hardware, with no method linked. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | The SIGMOD 2022 paper reports an average customer speedup of 3x over the previous Databricks Runtime, with a maximum over 10x. | [Photon paper, SIGMOD 2022](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago publishes operator microbenchmarks against NVIDIA cuDF (up to 10.5x on project, 8.2x on K-largest selection). An end-to-end Spark benchmark: none found on 2026-11-10. | [DataPelago Nucleus versus cuDF post](https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's June 2026 post says about 12x to 20x faster Spark task execution on a 100 GB Iceberg compaction; its home page shows 6x to 18x on a 110 GB enrichment pipeline with no method. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion, TPC-DS SF1000 on one i4i.8xlarge: Spark 3.5.5 135.6 min, Flarion 38.3 min, Photon 30.4 min. | [Flarion benchmarks page](https://www.flarion.io/benchmarks), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google reports 3.6x on TPC-H-like queries at 10 TB and, on its pricing page, up to 4.9x with no workload stated. | [Google Lightning Engine launch blog](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | A benchmark number for the Fabric native execution engine: none found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | TPC-DS 3 TB on 8 nodes of EMR on EKS: GPU 281 s, CPU clusters 1,010 to 1,043 s (AWS and NVIDIA). | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata's home page shows 62.7x on production Spark workloads, with no customer, data size or method. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which parts of your jobs stay on the old path? An accelerator only helps the operators it takes over. Jobs heavy in UDFs or RDDs spend most of their time on the path that did not change. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten falls back through columnar-to-row conversions; the docs list ANSI mode as always falling back and Parquet as the only fully supported scan format. | [Gluten Velox backend limitations](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks says Photon does not support UDFs, RDD APIs or Dataset APIs, and queries under about two seconds see no meaningful gain. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago tells users to avoid heavy use of RDDs and UDFs, and lists ANSI mode as running on the JVM. | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | A DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | [DualBird post, A Simpler Spark](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion says UDFs and UDAFs run on Spark as before, and the driver, planning, unsupported operators and often the final write still run on the JVM heap. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google says Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft says unsupported operators fall back to JVM Spark, and lists casting, timezone, round() and collect_list ordering differences on Runtime 1.3. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA says RDDs, GraphX and APIs that take Scala or Java functions are not accelerated, and unsupported operators fall back to the CPU. | [NVIDIA Spark plugin FAQ](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata says SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Can you predict the gain before you change the cluster? A pre-check on your own logs separates a TPC-DS result from your pipelines. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet marks fallback reasons in plan explain output, which shows what stays on Spark. | [Comet compatibility guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks tells users to check the Spark UI to see Photon fallback. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion offers a free assessment that simulates each job from logs and reports projected runtime and compute. | [Flarion pricing page](https://www.flarion.io/pricing), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents a qualification tool for checking jobs. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA documents a Qualification tool, a Profiling tool and an AutoTuner in its user guide. | [NVIDIA Spark plugin user guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata's Workload Analyzer predicts per-query acceleration before purchase from uploaded Spark logs. | [Speedata Workload Analyzer page](https://www.speedata.io/workload-analyzer), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Is the slow part compute, or shuffle, spill and I/O? Native engines speed up compute. A pipeline waiting on shuffle, small files or object storage may not move much. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron supports Celeborn 0.5 and 0.6 and Uniffle 0.10 as remote shuffle services. | [Apache Auron build script](https://raw.githubusercontent.com/apache/auron/master/auron-build.sh), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet supports remote shuffle through Celeborn. | [Comet Iceberg guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten supports Celeborn 0.3 to 0.5 and Uniffle 0.9.2 for remote shuffle. | [Gluten Velox Iceberg page](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks says queries under about two seconds see no meaningful gain from Photon. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird says tens of Spark settings become irrelevant, spills effectively disappear and skew stops causing spills. A later post says skew still costs time, but less. These are DualBird's claims. | [DualBird product page](https://dualbird.io/product?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google says I/O-bound or RDD-heavy jobs may suit the standard engine. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Can you rerun the benchmark yourself? A method that can be repeated on your own data can be checked; a number that can only be read cannot. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS publishes aws-samples/emr-spark-benchmark for a 3 TB TPC-DS run on OSS Spark and EMR; the repository states no results. | [aws-samples emr-spark-benchmark repository](https://github.com/aws-samples/emr-spark-benchmark), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | The Auron benchmark page links its data generator and runner tools and the full Spark configuration. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet publishes benchmark scripts, a data generation repository and its non-default Spark settings. | [Comet benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten ships the gluten-it tool for TPC-style runs; the homepage numbers do not link a run configuration. | [Gluten integration test tool README](https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's white paper has a landing page, and the full document is behind a form. No public repository was found. | [DualBird compaction white paper page](https://dualbird.io/resources/blog/%20spark-iceberg-compaction-benchmark-whitepaper?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion publishes its machine, builds, settings and caps, says the open-source engines can be rerun from public releases, and does not offer Flarion for download. | [Flarion benchmarks page](https://www.flarion.io/benchmarks), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA publishes NDS benchmark automation in a public Apache-2.0 repository and notes NDS is not comparable to published TPC-DS results. | [NVIDIA NDS benchmark README](https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata's customer tiles have no published method, and its white paper is behind a form. | [Speedata GitHub organization](https://github.com/Speedata-io), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - DataPelago Nucleus versus cuDF post: https://www.datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Flarion benchmarks page: https://www.flarion.io/benchmarks (read 10 Nov 2026) - Google Lightning Engine launch blog: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) - Gluten Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - DataPelago Accelerator for Spark page: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DualBird post, A Simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - NVIDIA Spark plugin FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Comet compatibility guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026) - Flarion pricing page: https://www.flarion.io/pricing (read 10 Nov 2026) - NVIDIA Spark plugin user guide: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - Speedata Workload Analyzer page: https://www.speedata.io/workload-analyzer (read 10 Nov 2026) - Apache Auron build script: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Comet Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Gluten Velox Iceberg page: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - DualBird product page: https://dualbird.io/product (read 10 Nov 2026) - aws-samples emr-spark-benchmark repository: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026) - Comet benchmarking guide: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - Gluten integration test tool README: https://raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md (read 10 Nov 2026) - DualBird compaction white paper page: https://dualbird.io/resources/blog/%20spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - NVIDIA NDS benchmark README: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026) - Speedata GitHub organization: https://github.com/Speedata-io (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/late-pipelines/ --- # What to check when AI training and feature pipelines wait on data preparation Situation, read 10 Nov 2026 **The short answer.** Feature engineering and training-data preparation are Spark jobs with wide tables and many UDFs. Speedata, DataPelago and DualBird each name AI data preparation among their uses, with claims of 100x, 10X and 10x to 30x, none of which is tied to an AI workload on a public page. ## 5 things to check, with the published facts that answer each ### Does the vendor name AI data preparation, and is a result attached? A use case on a home page is not a benchmark. Look for a named workload, data size and method. | Option | Published fact | Source, read | | --- | --- | --- | | [DataPelago](/options/datapelago/) | DataPelago's home page claims 10X faster at 80% cost reduction for GenAI and analytics data processing. No workload, baseline or method is attached. | [DataPelago home page](https://www.datapelago.ai/), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird lists AI data infrastructure (feature engineering and AI data prep) first among its use cases. Its one worked example with a published method is Iceberg compaction. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's published benchmark is TPC-DS at 3 TB with AWS (281 s on GPU). An AI data preparation benchmark: none found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata claims up to 100x for batch ETL and AI data preparation. A customer tile cites 62.7x on production Spark workloads for enterprise AI data preprocessing, with no customer name, data size or method. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Do your preparation jobs use UDFs, RDDs or custom types? Python UDFs and RDD code are common in feature pipelines, and most accelerators leave them on the CPU. | Option | Published fact | Source, read | | --- | --- | --- | | [Databricks Photon](/options/databricks-photon/) | Databricks says Photon does not support UDFs, RDD APIs or Dataset APIs. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago lists heavy use of RDDs and UDFs among workloads to avoid. | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird says workloads dominated by Python UDFs fall back to the CPU and do not see the same benefit yet. | [DualBird post, A Simpler Spark](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion says UDFs and UDAFs run on Spark as before. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google does not recommend Native Query Execution for RDD-heavy jobs, UDFs or most Spark ML libraries. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA says Dataset APIs with custom classes, flatMap, foreach, GraphX and RDDs are not accelerated. | [NVIDIA Spark plugin FAQ](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata delegates UDFs to the CPU while the rest of the query runs on the APU. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which file and table formats hold your feature data? Support differs by format and by read or write. Check the format your feature store writes. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron 8.0.0-incubating expands native scans for Iceberg, Hudi and Paimon. Delta Lake is not mentioned in the pages fetched. | [Apache Auron 8.0.0 release notes](https://auron.apache.org/archives/v8.0.0-incubating.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet accelerates Iceberg tables with Parquet data files; native Iceberg writes are off by default and Delta Lake support is experimental. | [Comet Iceberg guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten's Velox backend fully supports Parquet scans and ORC partly; Iceberg is read-only and Hudi covers copy-on-write tables only. | [Gluten Velox backend limitations](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks says Photon accelerates Delta Lake, Iceberg and Parquet writes. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists Iceberg, Delta Lake and Hudi tables and Parquet, Avro and JSON files. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google's NQE reads Parquet, ORC, Iceberg and Delta Lake. Byte columns in ORC and Parquet and Struct, Array and Map in Parquet are unsupported inputs. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft lists Parquet, Delta and CSV, including data reached through OneLake shortcuts. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What hardware does the accelerator add next to your training cluster? Feature pipelines often run next to GPU training. Adding a second accelerator type changes scheduling and cost. | Option | Published fact | Source, read | | --- | --- | --- | | [DataPelago](/options/datapelago/) | DataPelago's Nucleus engine targets CPU, GPU and FPGA. The DPA-S docs pages read do not state a GPU requirement. | [DataPelago technology page](https://www.datapelago.ai/technology), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird runs on AWS F2 instances with up to 8 AMD Virtex UltraScale+ FPGAs each. | [AWS EC2 F2 instance page](https://aws.amazon.com/ec2/instance-types/f2/), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion runs CPU-native on instances already in use, and contrasts that with GPU accelerators. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's plugin needs GPUs from Volta to Blackwell, with driver R525 or later and Linux glibc 2.28 or later. | [NVIDIA cuDF plugin download page](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata's C200 is a PCIe Gen 5 card, with a 2U server holding two cards. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### In which cloud does the data sit? An option documented for one cloud is not an option for data in another. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS documents the runtime on EMR on EC2, EKS and Serverless. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks documents Photon on AWS, Azure and Google Cloud within Databricks compute. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago documents AWS (EKS) and Google Cloud (Dataproc, GKE). | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird is AWS only; F2 is listed in eight AWS regions. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists AWS, Azure, GCP, on-premises and air-gapped use. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents Lightning Engine on Google Cloud only. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft documents the engine only inside Fabric on Azure, on OneLake data. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - DataPelago home page: https://www.datapelago.ai/ (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Speedata home page: https://www.speedata.io/ (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - DataPelago Accelerator for Spark page: https://www.datapelago.ai/dpa (read 10 Nov 2026) - DualBird post, A Simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - NVIDIA Spark plugin FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Apache Auron 8.0.0 release notes: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Comet Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Gluten Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - DataPelago technology page: https://www.datapelago.ai/technology (read 10 Nov 2026) - AWS EC2 F2 instance page: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/ai-data-prep/ --- # What to check when moving Spark from a managed platform to an open stack Situation, read 10 Nov 2026 **The short answer.** A team leaving Databricks or another managed platform for open Spark loses the engine bundled with it. Photon is documented only inside Databricks. This page sets out what each option needs from you instead: a built jar, off-heap memory settings, a license or a marketplace subscription. ## 5 things to check, with the published facts that answer each ### What does the bundled engine do that you would give up? Check whether the engine you rely on exists outside its platform at all. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS documents the EMR runtime only where EMR, EMR on EKS and EMR Serverless are available. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks documents Photon only as part of Databricks compute on AWS, Azure and Google Cloud, not as a standalone engine for open source Spark or EMR. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents Lightning Engine on Google Cloud only. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft documents the native execution engine only inside Fabric, and it does not run on AWS EMR or EKS. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which options run on your target platform? Open stack here means Spark on EMR, EKS, Dataproc, Kubernetes or your own servers. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron builds for Spark 3.0 to 4.2 on Linux images and macOS. EMR, EKS or Databricks support: none found on 2026-11-10. | [Apache Auron build script](https://raw.githubusercontent.com/apache/auron/master/auron-build.sh), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's jars bundle Linux native libraries for amd64 and arm64. The docs warn about cloud-provider Spark forks. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten documents Ubuntu 20.04/22.04 and CentOS 8 as build and test platforms and has storage pages for S3, GCS and ABFS. | [Apache Gluten downloads](https://gluten.apache.org/downloads/), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago documents EKS, Dataproc and GKE. | [DataPelago Google Cloud deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird documents Apache Spark, EMR and EKS on AWS F2 instances. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists Databricks (init scripts), EMR (bootstrap action), Dataproc, HDInsight, Spark on Kubernetes and on-premises. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS, Dataproc 2.1 to 2.3 and EMR on EKS. | [NVIDIA cuDF plugin download page](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What would your team operate that the platform used to? Open-source plugins move memory sizing, jar loading and fallback checks to you. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | You must build the Auron jar yourself (Rust nightly and a JDK), and the memory settings move off-heap into executor overhead. | [Apache Auron README](https://raw.githubusercontent.com/apache/auron/master/README.md), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's tuning guide says every deployment must configure how much memory Comet can use, through the off-heap pool and executor memory overhead. | [Comet tuning guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten needs off-heap memory enabled and sized, the columnar shuffle manager and binary jars loaded ahead of Spark's jars. | [Apache Gluten getting started](https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago's mandatory settings include the plugin class, subscription ID, license location, off-heap memory, the columnar shuffle manager and Netty options. | [DataPelago AWS deployment docs](https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion says existing Spark tuning carries over, and shows each stage as native or Spark in telemetry. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's guide covers executor and task counts, pinned memory, concurrent tasks per GPU and a procedure for GPU out-of-memory errors. | [NVIDIA Spark plugin user guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which license and contract applies? An Apache-2.0 project and a metered subscription behave differently in procurement. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron is Apache-2.0 under the Apache Software Foundation. | [Apache Auron homepage](https://auron.apache.org/), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet is Apache-2.0 under the Apache Software Foundation. | [Apache DataFusion Comet GitHub repository](https://api.github.com/repos/apache/datafusion-comet), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten is Apache-2.0. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Photon is proprietary to Databricks and its source is not open. | [Photon paper, SIGMOD 2022](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | DataPelago's Nucleus and DPA-S are proprietary and licensed, billed pay-as-you-go through the AWS and GCP marketplaces. | [DataPelago home page](https://www.datapelago.ai/), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird publishes no price, price list or marketplace listing. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion is proprietary with usage-based list prices. | [Flarion pricing page](https://www.flarion.io/pricing), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's source code is Apache-2.0 and its documentation CC-BY-4.0; you pay for the GPU instances. | [NVIDIA cuDF plugin license](https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE), 10 Nov 2026 | | [Speedata](/options/speedata/) | Speedata sells servers or goes through OEM or cloud partners, with no price list published. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which Spark versions move with you? On a managed platform the Spark version is the platform's runtime. On an open stack you choose it, and each plugin supports a list. | Option | Published fact | Source, read | | --- | --- | --- | | [Apache Auron](/options/apache-auron/) | Auron lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2. | [Apache Auron build script](https://raw.githubusercontent.com/apache/auron/master/auron-build.sh), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten names Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported. | [Gluten Velox backend guide](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks says the Spark version under Photon is the Databricks Runtime version, not a free choice of open source Spark. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DualBird](/options/dualbird/) | A DualBird list of supported Spark versions: none found on 2026-11-10. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion publishes no supported Spark version list; its benchmark ran Spark 3.5.5. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | The v26.10.0 release supports Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0. | [NVIDIA cuDF plugin download page](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - Apache Auron build script: https://raw.githubusercontent.com/apache/auron/master/auron-build.sh (read 10 Nov 2026) - Comet installation guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026) - Apache Gluten downloads: https://gluten.apache.org/downloads/ (read 10 Nov 2026) - DataPelago Google Cloud deployment docs: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026) - Apache Auron README: https://raw.githubusercontent.com/apache/auron/master/README.md (read 10 Nov 2026) - Comet tuning guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026) - Apache Gluten getting started: https://raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-started.md (read 10 Nov 2026) - DataPelago AWS deployment docs: https://datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html (read 10 Nov 2026) - Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026) - NVIDIA Spark plugin user guide: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026) - Apache Auron homepage: https://auron.apache.org/ (read 10 Nov 2026) - Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - Photon paper, SIGMOD 2022: https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf (read 10 Nov 2026) - DataPelago home page: https://www.datapelago.ai/ (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Flarion pricing page: https://www.flarion.io/pricing (read 10 Nov 2026) - NVIDIA cuDF plugin license: https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE (read 10 Nov 2026) - Speedata product page: https://www.speedata.io/product (read 10 Nov 2026) - Gluten Velox backend guide: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/Velox.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/leaving-managed/ --- # What to check when daily Iceberg compaction is a large part of the Spark bill Situation, read 10 Nov 2026 **The short answer.** A team paying for daily Iceberg compaction should check who ran any benchmark, whether its table matches theirs, what cost per terabyte includes, and where the job can run. Of the twelve options, one has a compaction benchmark: DualBird's, run by DualBird, method behind a sign-up form (checked 2026-11-10, the read date). ## 5 things to check, with the published facts that answer each ### Has anyone published a compaction benchmark, and who ran it? A compaction result run by the seller and described without scripts is a claim to verify. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | None found on 2026-11-10. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | An Auron compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS at 1 TB. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | A Comet compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's own figure, from its post of June 11, 2026 (the post date; 2026-11-10 is only the day it was read): sort-based Iceberg compaction on 100 GB, about 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark. DualBird ran it. The white paper's method is behind a form, and no public repository was found. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which options read and write Iceberg through the accelerator? Compaction is a write job. An accelerator that reads Iceberg but writes through vanilla Spark leaves the expensive step alone. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS lists Apache Iceberg v3 support and claims 2.7x faster Iceberg write performance, with no workload stated. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | Auron 8.0.0-incubating added native scan support for Iceberg, Hudi and Paimon, including copy-on-write tables and Iceberg changelog scans. | [Apache Auron 8.0.0 release notes](https://auron.apache.org/archives/v8.0.0-incubating.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet has a native Iceberg reader on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11 on Parquet data files. Native writes are off by default (spark.comet.write.iceberg.enabled). | [Comet Iceberg guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten's Iceberg page says only reading is supported, and only Spark 3.4 is well tested. | [Gluten Velox Iceberg page](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | Databricks says Photon accelerates Delta Lake, Apache Iceberg and Parquet writes. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | Iceberg appears in DualBird's compaction benchmark and in the meta description of its December 2025 demo post. A table format support list: none found on 2026-11-10. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | Flarion lists Iceberg, Delta Lake and Hudi table formats. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google's Native Query Execution reads Parquet, ORC, Apache Iceberg and Delta Lake. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's user guide has Iceberg and Delta Lake support pages, including reading and writing tables. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What does compaction cost per terabyte, and what is included? Cost per terabyte is the number that compares to your current bill, if the instance, storage and license costs are all in it. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS publishes no cost figure for the Iceberg write claim: none found on 2026-11-10. EMR price is added to EC2 and EBS cost. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | None found on 2026-11-10. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | A cost-per-job result for Comet: none found on 2026-11-10. | [Comet benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | A cost-per-job result for Gluten: none found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's own figure, baseline instance types not stated: about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. DualBird attributes part of the saving to F2 local NVMe replacing EBS volumes. The plugin's pricing model is not stated. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Does the test table look like yours? Table width, row count, compression and sort order are the inputs a compaction result depends on, so match them before trusting a ratio. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | None found on 2026-11-10. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | Auron's TPC-DS 1 TB data is Parquet with zstd; node types and counts are not given. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's Parquet data in its TPC-DS run is zstd compressed at scale factor 1000, on r6i instances with EBS. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | DualBird's own test dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3, 100 GB. Instance types and counts, and Spark version, are not stated in the public post. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Where can the compaction job run? A daily job needs capacity every day, in the region and cloud that holds the tables. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS documents the EMR runtime on EMR on EC2, EMR on EKS and EMR Serverless. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Apache Auron](/options/apache-auron/) | None found on 2026-11-10. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 | | [Apache DataFusion Comet](/options/datafusion-comet/) | None found on 2026-11-10. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 | | [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 | | [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 | | [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 | | [DualBird](/options/dualbird/) | F2 instances are listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2. DualBird is AWS only. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 | | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents Lightning Engine on Google Cloud only. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 | | [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft documents the engine only inside Fabric on Azure. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 | | [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's AWS test ran in us-east-1 on g7.4xlarge instances. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 | | [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026) - Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026) - Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026) - DataPelago DPA page: https://www.datapelago.ai/dpa (read 10 Nov 2026) - Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026) - DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Flarion product page: https://www.flarion.io/product (read 10 Nov 2026) - Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026) - Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026) - AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026) - Speedata technology page: https://www.speedata.io/technology (read 10 Nov 2026) - Apache Auron 8.0.0 release notes: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026) - Comet Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026) - Gluten Velox Iceberg page: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Comet benchmarking guide: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026) - AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/iceberg-compaction/ --- # The Ledger, the Change Budget and the option facts as downloads Datasets, read 10 Nov 2026 **The short answer.** Downloadable data behind the site: the options, the Spark Accelerator Ledger of 21 published benchmarks and the Change Budget of what each option makes a team change. Every row carries its source link and read date, and numbers are quoted as published. ## Downloads | Dataset | Rows | CSV | JSON | | --- | --- | --- | --- | | Spark Accelerator Ledger | 21 | [spark-accelerator-ledger.csv](/datasets/spark-accelerator-ledger.csv) | [spark-accelerator-ledger.json](/datasets/spark-accelerator-ledger.json) | | Change Budget | 48 | [change-budget.csv](/datasets/change-budget.csv) | [change-budget.json](/datasets/change-budget.json) | | Option facts | 132 | [option-facts.csv](/datasets/option-facts.csv) | [option-facts.json](/datasets/option-facts.json) | Edition 10 Nov 2026. License CC BY 4.0. ### Spark Accelerator Ledger Every published benchmark found for the 12 options on this site, with who ran it, on what and at what cost, as published. Ordered by publication date, newest first. ### Change Budget What each option makes a team change, in four columns: code, cluster, cloud and vendor. Alphabetical by option. ### Option facts Every published fact behind the option profiles, with its topic, source URL and read date. Alphabetical by option. ## Cite this Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0). Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/datasets/