Skip to content
Spark Acceleration Compare

Situation, read 10 Nov 2026

What to check when the Spark bill on EMR or EKS has doubled

All situationsOpen the Ledger

5 things to check, with the published facts that answer each

Which options publish a price or a cost per run, and what does it leave out?

A speed claim does not become a bill until the price of the plugin, the instances and the platform are all on the page.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS says the EMR price is added to the EC2 and EBS price, and EMR on EKS is charged per vCPU-hour and GB-hour requested plus EKS and EC2 or Fargate cost.Amazon EMR pricing, 10 Nov 2026
Apache AuronA cost-per-job result for Auron: none found on 2026-11-10. The project's only resource claim is about 50% fewer cluster resources.Apache Auron homepage, 10 Nov 2026
Apache DataFusion CometA cost-per-job result for Comet: none found on 2026-11-10.Comet benchmarking guide, 10 Nov 2026
Apache Gluten with VeloxA cost-per-job result for Gluten: none found on 2026-11-10. The software is free under Apache-2.0.Apache Gluten homepage, 10 Nov 2026
DataPelagoDataPelago describes pay-as-you-go billing through the AWS and GCP marketplaces; no unit price is published on the pages read.DataPelago Accelerator for Spark page, 10 Nov 2026
DualBirdDualBird publishes percentages (50% to 90% lower costs, 55% to 85% lower EC2 cost) and one figure of about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. No price for the plugin is published, so it is not stated whether the F2 instance price or a license fee is inside the savings.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionFlarion lists $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP and $0.012 for fully managed Spark such as EMR, billed on accelerated Spark cores.Flarion pricing page, 10 Nov 2026
Google Cloud Lightning EngineGoogle lists $0.06 per DCU-hour standard and $0.089 premium (Lightning Engine) for serverless in us-central1; clusters add a management fee, a Lightning Engine add-on and Compute Engine costs.Google Managed Service for Apache Spark pricing, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkAWS reports $2.06 per TPC-DS 3 TB run on the GPU cluster against $2.93 to $3.18 on CPU clusters, using On-Demand us-east-1 prices as of May 2026. The cost excludes the EKS control-plane fee, EBS, S3 and the driver pod.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata publishes no price list; its claims are up to 90% lower TCO and 37 servers replaced by 3, and hardware must be bought or rented.Speedata product page, 10 Nov 2026

Does it run on the EMR or EKS setup you already have?

An option that needs a different runtime or a different Spark build changes what your jobs run on, not just what they run with.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS lists the runtime on EMR on EC2, EMR on EKS and EMR Serverless, and calls it 100% API compatible with open source Spark.Amazon EMR Spark features page, 10 Nov 2026
Apache AuronAuron's docs describe a jar on the Spark classpath. Documented support or a tested setup for Amazon EMR, EKS or Databricks: none found on 2026-11-10.Apache Auron README, 10 Nov 2026
Apache DataFusion CometComet's TPC-DS run used an EKS cluster on r6i instances. The docs warn it may not fully work with cloud-provider Spark forks, so EMR is not documented as supported.Comet installation guide, 10 Nov 2026
Apache Gluten with VeloxGluten's docs do not mention EMR, EKS or Databricks support; the stated official Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1.Gluten Velox backend guide, 10 Nov 2026
DataPelagoDataPelago ships DPA-S on AWS as a Docker container in ECR for OSS Spark on EKS (Spark 3.5.1, Java 11, Scala 2.12). EMR and Databricks are not mentioned on the AWS docs page.DataPelago AWS deployment docs, 10 Nov 2026
DualBirdDualBird's home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. Spark version numbers and table format support lists: none found on 2026-11-10.DualBird home page, 10 Nov 2026
FlarionFlarion lists Amazon EMR (bootstrap action) and Spark on Kubernetes (Helm chart or Spark operator) among its platforms, and publishes no supported Spark version list.Flarion product page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkAmazon EMR on EKS supports the plugin natively from release 6.9.0, and the '-spark-rapids' release variant ships it pre-installed.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata says Dash accelerates Spark 3.x on Kubernetes, YARN and standalone managers. EMR and EKS support is not stated on the pages read.Speedata technology page, 10 Nov 2026

What instance or hardware change comes with it?

The bill moves with the instance family. A change to GPU or FPGA instances is a capacity and availability question as well as a price one.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkUsing the EMR runtime means running on EMR (on EC2, EKS or Serverless); AWS states no instance family change.Amazon EMR Spark features page, 10 Nov 2026
Apache DataFusion CometComet's Maven Central jars bundle native libraries for Linux on amd64 and arm64. No GPU or special hardware is required.Comet installation guide, 10 Nov 2026
DualBirdDualBird needs AWS EC2 F2 instances: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs) and f2.48xlarge (8 FPGAs, 192 vCPU). F2 is listed in eight regions, not including us-east-2 or eu-west-1.AWS EC2 instance types by Region, 10 Nov 2026
FlarionFlarion says it runs CPU-native on the fleet already in use, and contrasts that with GPU accelerators.Flarion FAQ, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkThe AWS test used 8 x g7.4xlarge at $26.35 per hour for the cluster, against $10.43 to $10.99 per hour for the CPU clusters.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata's C200 is a PCIe Gen 5 card in a physical server. The AWS option runs the work in a Speedata-hosted environment that syncs data from S3.Speedata home page, 10 Nov 2026

Is the baseline in the benchmark the Spark you run?

Speedups are ratios. A slow or unusual baseline inflates the ratio without changing your bill.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS says up to 4.5x faster than open-source equivalents; the page does not state the TPC-DS scale or cluster.Amazon EMR Spark features page, 10 Nov 2026
Apache DataFusion CometThe Comet project's own page shows the baseline matters: Spark 4.2.0 took 994 s and Spark 4.1.3 took 1,336 s on the same cluster, so Comet 1.2.0 is published as 2.0x and 2.51x against them (488 s and 532 s). The project attributes the gap mostly to the default S3A input stream.Comet TPC-DS benchmark results, 10 Nov 2026
DualBirdDualBird's baseline wording varies: the compaction post compares with vanilla Spark and an unnamed C++ accelerated Spark, while an April 2026 post says a high-end C++ columnar engine on a single CPU core.DualBird home page, 10 Nov 2026
FlarionFlarion ran Spark 3.5.5 as its baseline on one i4i.8xlarge: 135.6 min at 1 TB against 38.3 min for Flarion.Flarion benchmarks page, 10 Nov 2026
Google Cloud Lightning EngineGoogle compares with open source Spark on similar infrastructure (3.6x on TPC-H-like queries at 10 TB) and publishes no scripts, cluster spec or raw timings.Google Lightning Engine launch blog, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkAWS compares GPU instances with three CPU instance families (m9gd, m8id, m8a) at 3 TB; NVIDIA's own page notes its EC2 comparison uses different hardware and software stacks.NVIDIA Spark plugin overview, 10 Nov 2026

Can you test one job and switch it off?

A trial on a single nightly job produces a cost figure for your own workload, which no published benchmark does.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron describes fine-grained fallback for unsupported operators, and its Spark UI has an Auron tab that records per-operator fallback reasons.Apache Auron homepage, 10 Nov 2026
Apache DataFusion CometComet's plan explain output marks fallback reasons with [COMET: ...].Comet compatibility guide, 10 Nov 2026
FlarionFlarion says you can turn it off with one config change to compare, and offers a free assessment that simulates each job from logs.Flarion FAQ, 10 Nov 2026
Google Cloud Lightning EngineGoogle documents a qualification tool for checking jobs before enabling the engine at cluster creation.Google Lightning Engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkAWS recommends a pre-migration assessment with the NVIDIA qualification tool and spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on the CPU.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata offers a free Workload Analyzer: upload Spark logs, run a CLI locally or compare against TPC-DS.Speedata Workload Analyzer page, 10 Nov 2026

Sources

  1. Amazon EMR pricing, read 10 Nov 2026
  2. Apache Auron homepage, read 10 Nov 2026
  3. Comet benchmarking guide, read 10 Nov 2026
  4. Apache Gluten homepage, read 10 Nov 2026
  5. DataPelago Accelerator for Spark page, read 10 Nov 2026
  6. DualBird Iceberg compaction benchmark post, read 10 Nov 2026
  7. Flarion pricing page, read 10 Nov 2026
  8. Google Managed Service for Apache Spark pricing, read 10 Nov 2026
  9. AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
  10. Speedata product page, read 10 Nov 2026
  11. Amazon EMR Spark features page, read 10 Nov 2026
  12. Apache Auron README, read 10 Nov 2026
  13. Comet installation guide, read 10 Nov 2026
  14. Gluten Velox backend guide, read 10 Nov 2026
  15. DataPelago AWS deployment docs, read 10 Nov 2026
  16. DualBird home page, read 10 Nov 2026
  17. Flarion product page, read 10 Nov 2026
  18. Speedata technology page, read 10 Nov 2026
  19. AWS EC2 instance types by Region, read 10 Nov 2026
  20. Flarion FAQ, read 10 Nov 2026
  21. Speedata home page, read 10 Nov 2026
  22. Comet TPC-DS benchmark results, read 10 Nov 2026
  23. Flarion benchmarks page, read 10 Nov 2026
  24. Google Lightning Engine launch blog, read 10 Nov 2026
  25. NVIDIA Spark plugin overview, read 10 Nov 2026
  26. Comet compatibility guide, read 10 Nov 2026
  27. Google Lightning Engine documentation, read 10 Nov 2026
  28. Speedata Workload Analyzer page, read 10 Nov 2026

Last reviewed 10 Nov 2026.