# Questions about speeding up Spark, answered from published facts

FAQ, read 10 Nov 2026

**The short answer.** Thirty questions a data engineer asks about speeding up Spark without rewriting jobs, answered with sourced facts: what no code change means, which clouds each option supports, how benchmarks are run, what fallback costs, where prices are published and what none found means.

## 30 questions and answers

**What does each kind of Spark accelerator actually change?**

Plugins and native engines (Apache Auron, Apache Gluten, DataFusion Comet, Flarion) swap Spark's JVM execution for C++ or Rust code behind a spark.plugins setting. Platform runtimes (Databricks Photon, the Amazon EMR runtime, Google's Lightning Engine, Fabric's native engine) are built into one cloud service. Hardware options (NVIDIA RAPIDS, DualBird, Speedata) add GPUs, FPGAs or a card. See the [Change Budget](/change-budget/) for code, cluster, cloud and vendor changes.

**Which options run on Amazon EMR or EMR on EKS?**

Per each vendor's own pages: DataPelago documents Spark on EKS, DualBird says it is compatible with Amazon EMR and EKS, Flarion lists an EMR bootstrap action, and the NVIDIA RAPIDS Accelerator is supported on EMR on EKS from release 6.9.0. The Amazon EMR runtime is AWS's own. Auron, Comet and Gluten document no EMR support; none found on 2026-11-10.

**Which options are open source?**

Apache Auron, Apache Gluten and DataFusion Comet are Apache-2.0 projects. The NVIDIA RAPIDS Accelerator source is Apache-2.0 and its documentation is CC-BY-4.0. Databricks Photon, the Amazon EMR runtime, DataPelago, Flarion and Speedata are proprietary. For DualBird, no public repository was found on 2026-11-10. Google's Lightning Engine and Fabric's engine are cloud services, though Fabric's builds on Velox and Gluten.

**Which options need GPUs or special instances?**

The NVIDIA RAPIDS Accelerator needs NVIDIA GPUs (Volta through Blackwell generations). DualBird needs AWS F2 FPGA instances. Speedata needs its C200 APU card in a physical server or its hosted service. Google's native query execution can run on Axion ARM (C4A) machines but also on x86. Comet, Flarion and Gluten run on standard CPU instances. DataPelago's DPA-S docs state no GPU requirement, though its engine targets CPU, GPU and FPGA.

**How does operator fallback work?**

When a plugin cannot run an operator or expression natively, Spark runs that part on the JVM and the plugin runs the rest. The job returns a result, but the fallback stage gets no speedup, and data may be converted between columns and rows. NVIDIA says unsupported operators and UDFs fall back to the CPU. Gluten documents fallback through columnar-to-row conversions.

**How should I read a Spark acceleration benchmark?**

Check who ran it, the baseline Spark version and settings, the data size or scale factor, the number of queries and whether scripts are published. A vendor's own run is the vendor's claim. NVIDIA, for example, notes its NDS comparison used different hardware and software stacks. The [Ledger](/ledger/) lists these fields for every published entry we found.

**Why are most Spark benchmark results not official TPC results?**

The TPC audits results and publishes a report for each. Most Spark accelerator tests are runs by a vendor or project using TPC-derived queries, written TPC-DS-like, TPC-H-like or NDS. NVIDIA's own disclaimer on NDS is one example of that labeling. Databricks's audited 100 TB TPC-DS result of November 2021 ran on Databricks SQL, not a Spark batch job.

**What is DualBird and what does it require?**

DualBird says its Spark plugin runs sort, join and shuffle work on FPGAs in AWS F2 instances. DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published.

**What does DualBird claim about speed and cost?**

DualBird says 10x-30x faster data processing with 50%-90% lower costs and no changes required. The meta description on the same page says 10x-100x. For Iceberg compaction on 100 GB DualBird reports roughly 12-20x faster single-task execution and about $2 per terabyte against about $6 per terabyte, rising past $13, for CPU clusters. DualBird ran these tests itself.

**What does DualBird say a team has to change?**

DualBird says onboarding is two steps: change the EC2 instance type to F2 and add its Spark plugin. It also says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. DualBird publishes no Spark version list; none found on 2026-11-10.

**Which AWS F2 sizes exist?**

AWS lists three F2 sizes: f2.6xlarge with 1 FPGA, 24 vCPU and 256 GiB of memory; f2.12xlarge with 2 FPGAs, 48 vCPU and 512 GiB; and f2.48xlarge with 8 FPGAs, 192 vCPU and 2,048 GiB. Each FPGA is an AMD Virtex UltraScale+ HBM VU47P with 16 GiB of high-bandwidth memory.

**What does AWS say its own EMR Spark runtime does?**

AWS says the performance-optimized EMR runtime executes up to 4.5x faster than open source equivalents, and writes to Iceberg up to 2.7x faster. The page gives no TPC-DS scale or cluster for those figures. AWS publishes a benchmark repository, aws-samples/emr-spark-benchmark, with a 3 TB TPC-DS dataset and the commands to run it yourself.

**What did AWS and NVIDIA publish for GPU Spark on EMR on EKS?**

AWS reports TPC-DS at 3 TB on 8-node EMR on EKS clusters: 281 seconds on g7.4xlarge GPU instances against 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. The test used Spark 3.5.6, the median of 5 iterations, in us-east-1. NVIDIA's own overview adds separate NDS runs on different hardware.

**What did the GPU test cost, from published prices?**

AWS reports $2.06 per benchmark run on the GPU cluster against $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The GPU cluster cost $26.35 per hour (8 x $3.042 for EC2 plus $2.01 for EMR on EKS); the CPU clusters cost $10.43 to $10.99 per hour. AWS counts neither the EKS control-plane fee, EBS, S3 nor the driver pod.

**How does AWS charge for EMR on EKS?**

AWS charges EMR on EKS by the vCPU and memory a job requests, from image download until the pod ends, rounded up to the nearest second, added to EKS and compute costs. Its pricing example: 100 vCPU and 300 GB for 30 minutes at $0.007084 per vCPU-hour and $0.000777875 per GB-hour is a $0.4709 EMR uplift. Each EKS cluster also costs $0.10 per hour.

**What does Google's Lightning Engine cost?**

Google's pricing page gives example us-central1 rates for serverless Managed Service for Apache Spark: $0.06 per DCU-hour on the standard tier and $0.089 per DCU-hour on the premium tier that includes Lightning Engine. Billing is per second with a one-minute minimum. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. It runs on Google Cloud only.

**What does Flarion charge?**

Flarion publishes usage-based list prices: $0.009 per CPU hour for self-hosted Spark on AWS, Azure or GCP, $0.012 per CPU hour for fully managed Spark such as EMR, $0.095 per DBU on Databricks, and $0.75 per CPU per month on premises. Billing is on accelerated Spark cores, driver plus executors. That is the software fee only; cluster costs are separate.

**Are prices published for DualBird, DataPelago, Speedata and Photon?**

DualBird: none found on 2026-11-10. Speedata: no price list; the pages offer a server purchase or partner routes. DataPelago: pay-as-you-go billing through the AWS and GCP marketplaces, with no unit price on the pages read. Databricks Photon: instance types use DBUs at a different rate than non-Photon, so the DBU multiplier on Databricks's pricing page sets the cost.

**Which Spark versions do the open source plugins support?**

From each project's own documents: Apache Auron lists Spark 3.0 to 3.5 and 4.0 to 4.2 in its build script; Apache Gluten officially supports 3.4.4, 3.5.5, 4.0.2 and 4.1.1; DataFusion Comet supports 3.5.9, 4.0.4, 4.1.3 and 4.2.0; the NVIDIA RAPIDS Accelerator v26.10.0 supports 3.3.x through 4.2.0. Check your EMR release label's Spark build before testing.

**Which options document Apache Iceberg support?**

DataFusion Comet has a native Iceberg reader tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11 and native writes that are off by default. Apache Auron added native scans for Iceberg, Hudi and Paimon in 8.0.0-incubating. Gluten's Velox backend reads Iceberg tables only. Databricks says Photon accelerates Iceberg writes. DualBird mentions Iceberg in its own compaction benchmark and in the description of a December 2025 demo post.

**Do any options work on Databricks or Microsoft Fabric?**

Databricks Photon is built into Databricks compute. Microsoft's Fabric native execution engine runs only inside Fabric, on Runtime 1.3 (Spark 3.5) and Runtime 2.0 (Spark 4.1). Flarion lists Databricks init scripts, and the NVIDIA RAPIDS Accelerator lists Databricks runtimes 14.3 ML LTS and 17.3 ML LTS. DualBird is AWS F2 only.

**Do I need to change memory settings after adding a plugin?**

Usually yes. Gluten's quick start sets spark.memory.offHeap.size=20g. Comet's benchmark used 32G off-heap with 64G executors. Auron instead disables Spark off-heap memory and raises executor overhead, 4096 MB in its quick start. Spark's off-heap size defaults to 0 and must be positive when off-heap is enabled. Treat memory sizing as part of the trial.

**Do UDFs and RDD code get accelerated?**

Mostly no. NVIDIA says UDFs fall back to the CPU and RDDs are not accelerated. Databricks's Photon page lists UDFs, RDD APIs and Dataset APIs as unsupported. Google does not recommend its native query execution for RDD-heavy jobs or UDFs. DataPelago advises avoiding heavy RDD and UDF use. Flarion says UDFs and UDAFs keep running on Spark as before.

**Are results identical to plain Spark?**

Not always. NVIDIA aims for bit-for-bit identical results and documents operator compatibility. Comet leaves some natively different expressions off unless allowIncompatible is set. Gluten's regexp functions use RE2, which has no lookaround, and ANSI mode always falls back. DataPelago warns of possible incorrect results in case-sensitive mode. Compare output on your own data first.

**Can I test an option before changing production?**

Several publish tools. NVIDIA has a Qualification tool and an AutoTuner, Google has a qualification tool for native query execution, Speedata offers a free Workload Analyzer that reads Spark logs, and Flarion offers a free assessment that simulates each job from logs. Flarion also says one config change switches it off to compare.

**What does Speedata's APU require?**

Speedata says its Dash software plugs into Spark's Catalyst optimizer and offloads compute-heavy work to its APU, with UDFs delegated to the CPU. The APU is a PCIe Gen 5 card (C200), so it needs physical servers: a 2U server from Speedata, servers from Dell or HPE, or Speedata's hosted service for AWS. Speedata's pages state no EMR, EKS or Databricks support; none found on 2026-11-10.

**What does the Spark Accelerator Ledger list?**

The [Ledger](/ledger/) lists each published benchmark for each option: the workload, data size, hardware, result, cost where published, who ran it and whether they are the vendor, and whether the method is public. It lists only numbers a named party published, with the link and date. No number is averaged or estimated. Read the [Ledger method](/ledger/method/) for the entry rules.

**What does the Change Budget show?**

The [Change Budget](/change-budget/) shows, for each option, what a team has to change: code, cluster, cloud or vendor. Many options need no job code changes but do change the cluster, such as the instance type, memory settings or a plugin jar. Each cell links to the vendor's or project's own page.

**Does Adaptive Query Execution already do this work?**

No, AQE and accelerators work at different layers. AQE re-plans a query at runtime from statistics: it coalesces small shuffle partitions, splits skewed ones and can turn a sort-merge join into a broadcast join. It is on by default since Spark 3.2.0. Accelerators change how the chosen plan executes. EMR documents AQE and dynamic partition pruning as part of its own runtime.

**What does a Spark plugin install actually involve?**

Usually a jar on the classpath, the spark.plugins setting, memory changes and, for some, a custom shuffle manager. Gluten uses org.apache.gluten.GlutenPlugin and ColumnarShuffleManager. Speedata uses spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. Spark documents the plugin interface as a DeveloperApi, so check the supported Spark versions first.

## Sources

- Apache Gluten home page: https://gluten.apache.org/ (read 10 Nov 2026)
- AWS Big Data Blog: GPU-accelerated Apache Spark with Amazon EMR: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026)
- NVIDIA cudf-spark repository LICENSE: https://raw.githubusercontent.com/NVIDIA/cudf-spark/main/LICENSE (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark: download and requirements: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026)
- NVIDIA Spark RAPIDS user guide: overview: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026)
- NVIDIA NDS benchmark README: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026)
- DualBird home page: https://dualbird.io/ (read 10 Nov 2026)
- DualBird blog: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026)
- DualBird blog: A simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026)
- AWS: Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026)
- AWS: Amazon EMR features for Apache Spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026)
- AWS: Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026)
- Google Cloud: Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)
- Flarion: pricing: https://www.flarion.io/pricing (read 10 Nov 2026)
- Databricks: pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026)
- Apache DataFusion Comet: installation guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Microsoft Learn: Native execution engine for Fabric Spark: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026)
- Apache Spark configuration reference: https://spark.apache.org/docs/latest/configuration.html (read 10 Nov 2026)
- NVIDIA Spark RAPIDS user guide: FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026)
- Apache Gluten: Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026)
- Speedata: Workload Analyzer: https://www.speedata.io/workload-analyzer (read 10 Nov 2026)
- Speedata: product page: https://www.speedata.io/product (read 10 Nov 2026)
- Apache Spark documentation: SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html (read 10 Nov 2026)
- Apache Spark 4.3.0 API: SparkPlugin: https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/plugin/SparkPlugin.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/faq/
