Situation, read 10 Nov 2026
What to check when the Spark bill on EMR or EKS has doubled
5 things to check, with the published facts that answer each
Which options publish a price or a cost per run, and what does it leave out?
A speed claim does not become a bill until the price of the plugin, the instances and the platform are all on the page.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS says the EMR price is added to the EC2 and EBS price, and EMR on EKS is charged per vCPU-hour and GB-hour requested plus EKS and EC2 or Fargate cost. | Amazon EMR pricing, 10 Nov 2026 |
| Apache Auron | A cost-per-job result for Auron: none found on 2026-11-10. The project's only resource claim is about 50% fewer cluster resources. | Apache Auron homepage, 10 Nov 2026 |
| Apache DataFusion Comet | A cost-per-job result for Comet: none found on 2026-11-10. | Comet benchmarking guide, 10 Nov 2026 |
| Apache Gluten with Velox | A cost-per-job result for Gluten: none found on 2026-11-10. The software is free under Apache-2.0. | Apache Gluten homepage, 10 Nov 2026 |
| DataPelago | DataPelago describes pay-as-you-go billing through the AWS and GCP marketplaces; no unit price is published on the pages read. | DataPelago Accelerator for Spark page, 10 Nov 2026 |
| DualBird | DualBird publishes percentages (50% to 90% lower costs, 55% to 85% lower EC2 cost) and one figure of about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. No price for the plugin is published, so it is not stated whether the F2 instance price or a license fee is inside the savings. | DualBird Iceberg compaction benchmark post, 10 Nov 2026 |
| Flarion | Flarion lists $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP and $0.012 for fully managed Spark such as EMR, billed on accelerated Spark cores. | Flarion pricing page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google lists $0.06 per DCU-hour standard and $0.089 premium (Lightning Engine) for serverless in us-central1; clusters add a management fee, a Lightning Engine add-on and Compute Engine costs. | Google Managed Service for Apache Spark pricing, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | AWS reports $2.06 per TPC-DS 3 TB run on the GPU cluster against $2.93 to $3.18 on CPU clusters, using On-Demand us-east-1 prices as of May 2026. The cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata publishes no price list; its claims are up to 90% lower TCO and 37 servers replaced by 3, and hardware must be bought or rented. | Speedata product page, 10 Nov 2026 |
Does it run on the EMR or EKS setup you already have?
An option that needs a different runtime or a different Spark build changes what your jobs run on, not just what they run with.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS lists the runtime on EMR on EC2, EMR on EKS and EMR Serverless, and calls it 100% API compatible with open source Spark. | Amazon EMR Spark features page, 10 Nov 2026 |
| Apache Auron | Auron's docs describe a jar on the Spark classpath. Documented support or a tested setup for Amazon EMR, EKS or Databricks: none found on 2026-11-10. | Apache Auron README, 10 Nov 2026 |
| Apache DataFusion Comet | Comet's TPC-DS run used an EKS cluster on r6i instances. The docs warn it may not fully work with cloud-provider Spark forks, so EMR is not documented as supported. | Comet installation guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten's docs do not mention EMR, EKS or Databricks support; the stated official Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1. | Gluten Velox backend guide, 10 Nov 2026 |
| DataPelago | DataPelago ships DPA-S on AWS as a Docker container in ECR for OSS Spark on EKS (Spark 3.5.1, Java 11, Scala 2.12). EMR and Databricks are not mentioned on the AWS docs page. | DataPelago AWS deployment docs, 10 Nov 2026 |
| DualBird | DualBird's home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. Spark version numbers and table format support lists: none found on 2026-11-10. | DualBird home page, 10 Nov 2026 |
| Flarion | Flarion lists Amazon EMR (bootstrap action) and Spark on Kubernetes (Helm chart or Spark operator) among its platforms, and publishes no supported Spark version list. | Flarion product page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | Amazon EMR on EKS supports the plugin natively from release 6.9.0, and the '-spark-rapids' release variant ships it pre-installed. | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata says Dash accelerates Spark 3.x on Kubernetes, YARN and standalone managers. EMR and EKS support is not stated on the pages read. | Speedata technology page, 10 Nov 2026 |
What instance or hardware change comes with it?
The bill moves with the instance family. A change to GPU or FPGA instances is a capacity and availability question as well as a price one.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | Using the EMR runtime means running on EMR (on EC2, EKS or Serverless); AWS states no instance family change. | Amazon EMR Spark features page, 10 Nov 2026 |
| Apache DataFusion Comet | Comet's Maven Central jars bundle native libraries for Linux on amd64 and arm64. No GPU or special hardware is required. | Comet installation guide, 10 Nov 2026 |
| DualBird | DualBird needs AWS EC2 F2 instances: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs) and f2.48xlarge (8 FPGAs, 192 vCPU). F2 is listed in eight regions, not including us-east-2 or eu-west-1. | AWS EC2 instance types by Region, 10 Nov 2026 |
| Flarion | Flarion says it runs CPU-native on the fleet already in use, and contrasts that with GPU accelerators. | Flarion FAQ, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | The AWS test used 8 x g7.4xlarge at $26.35 per hour for the cluster, against $10.43 to $10.99 per hour for the CPU clusters. | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata's C200 is a PCIe Gen 5 card in a physical server. The AWS option runs the work in a Speedata-hosted environment that syncs data from S3. | Speedata home page, 10 Nov 2026 |
Is the baseline in the benchmark the Spark you run?
Speedups are ratios. A slow or unusual baseline inflates the ratio without changing your bill.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS says up to 4.5x faster than open-source equivalents; the page does not state the TPC-DS scale or cluster. | Amazon EMR Spark features page, 10 Nov 2026 |
| Apache DataFusion Comet | The Comet project's own page shows the baseline matters: Spark 4.2.0 took 994 s and Spark 4.1.3 took 1,336 s on the same cluster, so Comet 1.2.0 is published as 2.0x and 2.51x against them (488 s and 532 s). The project attributes the gap mostly to the default S3A input stream. | Comet TPC-DS benchmark results, 10 Nov 2026 |
| DualBird | DualBird's baseline wording varies: the compaction post compares with vanilla Spark and an unnamed C++ accelerated Spark, while an April 2026 post says a high-end C++ columnar engine on a single CPU core. | DualBird home page, 10 Nov 2026 |
| Flarion | Flarion ran Spark 3.5.5 as its baseline on one i4i.8xlarge: 135.6 min at 1 TB against 38.3 min for Flarion. | Flarion benchmarks page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google compares with open source Spark on similar infrastructure (3.6x on TPC-H-like queries at 10 TB) and publishes no scripts, cluster spec or raw timings. | Google Lightning Engine launch blog, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | AWS compares GPU instances with three CPU instance families (m9gd, m8id, m8a) at 3 TB; NVIDIA's own page notes its EC2 comparison uses different hardware and software stacks. | NVIDIA Spark plugin overview, 10 Nov 2026 |
Can you test one job and switch it off?
A trial on a single nightly job produces a cost figure for your own workload, which no published benchmark does.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron describes fine-grained fallback for unsupported operators, and its Spark UI has an Auron tab that records per-operator fallback reasons. | Apache Auron homepage, 10 Nov 2026 |
| Apache DataFusion Comet | Comet's plan explain output marks fallback reasons with [COMET: ...]. | Comet compatibility guide, 10 Nov 2026 |
| Flarion | Flarion says you can turn it off with one config change to compare, and offers a free assessment that simulates each job from logs. | Flarion FAQ, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google documents a qualification tool for checking jobs before enabling the engine at cluster creation. | Google Lightning Engine documentation, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | AWS recommends a pre-migration assessment with the NVIDIA qualification tool and spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on the CPU. | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata offers a free Workload Analyzer: upload Spark logs, run a CLI locally or compare against TPC-DS. | Speedata Workload Analyzer page, 10 Nov 2026 |
Sources
- Amazon EMR pricing, read 10 Nov 2026
- Apache Auron homepage, read 10 Nov 2026
- Comet benchmarking guide, read 10 Nov 2026
- Apache Gluten homepage, read 10 Nov 2026
- DataPelago Accelerator for Spark page, read 10 Nov 2026
- DualBird Iceberg compaction benchmark post, read 10 Nov 2026
- Flarion pricing page, read 10 Nov 2026
- Google Managed Service for Apache Spark pricing, read 10 Nov 2026
- AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- Amazon EMR Spark features page, read 10 Nov 2026
- Apache Auron README, read 10 Nov 2026
- Comet installation guide, read 10 Nov 2026
- Gluten Velox backend guide, read 10 Nov 2026
- DataPelago AWS deployment docs, read 10 Nov 2026
- DualBird home page, read 10 Nov 2026
- Flarion product page, read 10 Nov 2026
- Speedata technology page, read 10 Nov 2026
- AWS EC2 instance types by Region, read 10 Nov 2026
- Flarion FAQ, read 10 Nov 2026
- Speedata home page, read 10 Nov 2026
- Comet TPC-DS benchmark results, read 10 Nov 2026
- Flarion benchmarks page, read 10 Nov 2026
- Google Lightning Engine launch blog, read 10 Nov 2026
- NVIDIA Spark plugin overview, read 10 Nov 2026
- Comet compatibility guide, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- Speedata Workload Analyzer page, read 10 Nov 2026
Last reviewed 10 Nov 2026.