Change Budget, read 10 Nov 2026
The Change Budget: what each option makes a team change
Four columns, twelve options
| Option | Code | Cluster | Cloud | Vendor |
|---|---|---|---|---|
| Amazon EMR runtime for Apache Spark | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. |
| Apache Auron | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. |
| Apache DataFusion Comet | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. |
| Apache Gluten with Velox | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. |
| Databricks Photon | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. |
| DataPelago | No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs. | Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS. | AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account. | Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read. |
| DualBird | No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.' | Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird. | AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud. | Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated. |
| Flarion | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. |
| Google Cloud Lightning Engine | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Microsoft Fabric native execution engine | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. |
| NVIDIA cuDF plugin for Apache Spark | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. |
| Speedata | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. |
Download and cite
CSV: /datasets/change-budget.csv. JSON: /datasets/change-budget.json. Cite this: Spark Acceleration Compare, "Spark Accelerator Ledger" and "Change Budget", edition 10 Nov 2026, https://sparkaccelerationcompare.com/datasets/ (CC BY 4.0).
Sources
- Amazon EMR Spark features page, read 10 Nov 2026
- Amazon EMR Spark performance guide, read 10 Nov 2026
- Amazon EMR pricing, read 10 Nov 2026
- Apache Auron homepage, read 10 Nov 2026
- Apache Auron benchmark page, read 10 Nov 2026
- Apache Auron README, read 10 Nov 2026
- Comet compatibility guide, read 10 Nov 2026
- Comet tuning guide, read 10 Nov 2026
- Comet installation guide, read 10 Nov 2026
- Apache DataFusion Comet GitHub repository, read 10 Nov 2026
- Apache Gluten getting started, read 10 Nov 2026
- Apache Gluten downloads, read 10 Nov 2026
- Apache Gluten Powered By page, read 10 Nov 2026
- Databricks Photon documentation, read 10 Nov 2026
- Databricks pricing page, read 10 Nov 2026
- DataPelago Accelerator for Spark page, read 10 Nov 2026
- DataPelago AWS deployment docs, read 10 Nov 2026
- DataPelago Google Cloud deployment docs, read 10 Nov 2026
- DualBird about page, read 10 Nov 2026
- DualBird product page, read 10 Nov 2026
- AWS EC2 instance types by Region, read 10 Nov 2026
- DualBird Iceberg compaction benchmark post, read 10 Nov 2026
- Flarion FAQ, read 10 Nov 2026
- Flarion product page, read 10 Nov 2026
- Flarion pricing page, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- Google Managed Service for Apache Spark pricing, read 10 Nov 2026
- Fabric native execution engine documentation, read 10 Nov 2026
- Microsoft Fabric pricing, read 10 Nov 2026
- NVIDIA Spark plugin overview, read 10 Nov 2026
- AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
- NVIDIA cuDF plugin download page, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- Speedata home page, read 10 Nov 2026
Last reviewed 10 Nov 2026.