Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

What "no code change" has to mean

All guidesOpen the Ledger

Four things the phrase can cover

Vendors and projects use "no code change" for four different promises. Read the documentation for which one is being made before you compare two options on it.

  • Your job source is untouched. The same PySpark, Scala or SQL runs. This is true for every option below.
  • Your job runs accelerated only if it uses certain APIs. Spark SQL and the DataFrame API pass through Catalyst, which the accelerators hook into. RDD code, Dataset lambdas and most UDFs do not, so those stages run on the JVM unchanged and unaccelerated.
  • Your configuration changes. A plugin class, a shuffle manager, off-heap memory, executor overhead and a jar on the classpath. Your code is the same, your submit configuration is not.
  • Your infrastructure changes. A different instance family, a different runtime image, a different cloud service or a physical card. The job is the same and the bill is different.

SQL and DataFrame code versus RDDs and UDFs

Spark plans SQL and DataFrame operations as a tree of operators that a plugin can replace one by one. A Scala or Python function passed to map, flatMap, foreachPartition or a UDF is opaque to the planner. Databricks documents that Photon does not support UDFs, RDD APIs or Dataset APIs. DataPelago's documentation tells users to avoid heavy use of RDDs and UDFs. Google says Lightning Engine's native query execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NVIDIA's FAQ lists what its plugin does not accelerate: Dataset operations with custom classes or types, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs.

Pandas UDFs are a partial case. Spark runs them with Arrow transferring data to Pandas, which is vectorized on the Python side, but the function still runs in a Python worker, outside any native engine. A job that depends on them keeps that stage on CPU.

This is the first check for your own code: count how much of your runtime sits in stages built from UDFs and RDD operations. The guide on operator coverage and fallback shows how to find them.

Fallback: what happens when an operator is not covered

All the plugins and engines listed here run unsupported work on the stock Spark engine instead of failing the job. The documented mechanics differ.

Apache Auron describes fine-grained fallback and records per-operator reasons in an Auron tab in the Spark UI. Apache DataFusion Comet states that if a query stage contains an operator it does not support, that stage falls back to regular Spark execution, and results are unaffected. Apache Gluten converts between columnar and row formats at each boundary between native and JVM operators. Databricks says Photon falls back transparently. Flarion says operations it does not accelerate run on Spark automatically. Microsoft says Fabric's native execution engine falls back to JVM Spark. Speedata's page says UDFs and other SQL elements that cannot be accelerated are delegated to the CPU.

Each fallback has a cost: the conversion between formats, plus the lost speedup on that stage. A job that falls back in the middle of a plan can run slower than the baseline. None of the projects publishes a figure for that cost on a typical job: none found on 2026-11-10.

How each option defines it

Dates read 2026-11-10. The Change Budget page lists the same changes as code, cluster, cloud and vendor.
Option (alphabetical)What changes, per its documentation
Apache AuronJar built from source (Rust nightly and JDK), spark.auron.enable, a session extension and a shuffle manager; memory moved from off-heap to executor overhead
Apache DataFusion Cometspark.plugins, a shuffle manager and off-heap settings; some expressions are off unless allowIncompatible is set
Apache Gluten with Veloxspark.plugins=org.apache.gluten.GlutenPlugin, off-heap size, ColumnarShuffleManager and a bundle jar loaded ahead of Spark's jars; officially supported Spark versions 3.4.4, 3.5.5, 4.0.2 and 4.1.1
Databricks PhotonA cluster setting (runtime_engine PHOTON in the API); runs only on Databricks Runtime; DBU rate differs from the same instance without Photon
DataPelagoGluten plugin class, subscription ID and license bucket, off-heap size, columnar shuffle manager; Docker image on EKS or a Dataproc/GKE deployment
DualBirdPer DualBird: switch the EC2 instance type to F2 and add the DualBird Spark plugin; DualBird says no code changes
FlarionOne JAR in the existing cluster, installed through an init script, bootstrap action, Helm chart or operator; per Flarion, one config change to install and one to turn off
Google Lightning EngineCluster created with --engine=lightning on image 2.3.3 or later (or 3.0.2 or later); premium rate of $0.089 per DCU-hour versus $0.06 standard in the us-central1 example
Microsoft Fabric native execution engineToggle in an environment or spark.native.enabled true; Fabric only
NVIDIA cuDF plugin for Apache SparkPlugin enabled by a configuration setting and GPU instances; Spark 3.3.x to 4.2.0 across releases
Speedataspark.plugins=com.speedata.spark.DashPlugin, spark.speedata.apus=1 and a PCIe card in a physical server or a Speedata-hosted AWS service

Where DualBird's definition sits

DualBird says its plugin needs no code changes and that "the majority of common Spark performance configurations do not exist when using the solution". Its own engineer states that the FPGA path accelerates a defined set of operators (sort, join, aggregation and others) and that workloads dominated by Python UDFs or unsupported operators fall back to the CPU. Those are DualBird's statements, and DualBird does not publish a supported-operator list.

DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published.

Sources

  1. NVIDIA cuDF plugin for Apache Spark: FAQ, read 10 Nov 2026
  2. Databricks documentation: What is Photon, read 10 Nov 2026
  3. Google Cloud: Lightning Engine for Managed Service for Apache Spark, read 10 Nov 2026
  4. Google Cloud: Managed Service for Apache Spark pricing, read 10 Nov 2026
  5. Apache Spark documentation: Apache Arrow in PySpark, read 10 Nov 2026
  6. Apache Gluten: getting started, read 10 Nov 2026
  7. Apache Auron, read 10 Nov 2026
  8. Microsoft Learn: Native execution engine for Fabric Spark, read 10 Nov 2026
  9. Speedata product page, read 10 Nov 2026
  10. DualBird: A simpler Spark, read 10 Nov 2026
  11. Flarion FAQs, read 10 Nov 2026

Last reviewed 10 Nov 2026.