Guide, read 10 Nov 2026
What "no code change" has to mean
Four things the phrase can cover
Vendors and projects use "no code change" for four different promises. Read the documentation for which one is being made before you compare two options on it.
- Your job source is untouched. The same PySpark, Scala or SQL runs. This is true for every option below.
- Your job runs accelerated only if it uses certain APIs. Spark SQL and the DataFrame API pass through Catalyst, which the accelerators hook into. RDD code, Dataset lambdas and most UDFs do not, so those stages run on the JVM unchanged and unaccelerated.
- Your configuration changes. A plugin class, a shuffle manager, off-heap memory, executor overhead and a jar on the classpath. Your code is the same, your submit configuration is not.
- Your infrastructure changes. A different instance family, a different runtime image, a different cloud service or a physical card. The job is the same and the bill is different.
SQL and DataFrame code versus RDDs and UDFs
Spark plans SQL and DataFrame operations as a tree of operators that a plugin can replace one by one. A Scala or Python function passed to map, flatMap, foreachPartition or a UDF is opaque to the planner. Databricks documents that Photon does not support UDFs, RDD APIs or Dataset APIs. DataPelago's documentation tells users to avoid heavy use of RDDs and UDFs. Google says Lightning Engine's native query execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NVIDIA's FAQ lists what its plugin does not accelerate: Dataset operations with custom classes or types, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs.
Pandas UDFs are a partial case. Spark runs them with Arrow transferring data to Pandas, which is vectorized on the Python side, but the function still runs in a Python worker, outside any native engine. A job that depends on them keeps that stage on CPU.
This is the first check for your own code: count how much of your runtime sits in stages built from UDFs and RDD operations. The guide on operator coverage and fallback shows how to find them.
Fallback: what happens when an operator is not covered
All the plugins and engines listed here run unsupported work on the stock Spark engine instead of failing the job. The documented mechanics differ.
Apache Auron describes fine-grained fallback and records per-operator reasons in an Auron tab in the Spark UI. Apache DataFusion Comet states that if a query stage contains an operator it does not support, that stage falls back to regular Spark execution, and results are unaffected. Apache Gluten converts between columnar and row formats at each boundary between native and JVM operators. Databricks says Photon falls back transparently. Flarion says operations it does not accelerate run on Spark automatically. Microsoft says Fabric's native execution engine falls back to JVM Spark. Speedata's page says UDFs and other SQL elements that cannot be accelerated are delegated to the CPU.
Each fallback has a cost: the conversion between formats, plus the lost speedup on that stage. A job that falls back in the middle of a plan can run slower than the baseline. None of the projects publishes a figure for that cost on a typical job: none found on 2026-11-10.
How each option defines it
| Option (alphabetical) | What changes, per its documentation |
|---|---|
| Apache Auron | Jar built from source (Rust nightly and JDK), spark.auron.enable, a session extension and a shuffle manager; memory moved from off-heap to executor overhead |
| Apache DataFusion Comet | spark.plugins, a shuffle manager and off-heap settings; some expressions are off unless allowIncompatible is set |
| Apache Gluten with Velox | spark.plugins=org.apache.gluten.GlutenPlugin, off-heap size, ColumnarShuffleManager and a bundle jar loaded ahead of Spark's jars; officially supported Spark versions 3.4.4, 3.5.5, 4.0.2 and 4.1.1 |
| Databricks Photon | A cluster setting (runtime_engine PHOTON in the API); runs only on Databricks Runtime; DBU rate differs from the same instance without Photon |
| DataPelago | Gluten plugin class, subscription ID and license bucket, off-heap size, columnar shuffle manager; Docker image on EKS or a Dataproc/GKE deployment |
| DualBird | Per DualBird: switch the EC2 instance type to F2 and add the DualBird Spark plugin; DualBird says no code changes |
| Flarion | One JAR in the existing cluster, installed through an init script, bootstrap action, Helm chart or operator; per Flarion, one config change to install and one to turn off |
| Google Lightning Engine | Cluster created with --engine=lightning on image 2.3.3 or later (or 3.0.2 or later); premium rate of $0.089 per DCU-hour versus $0.06 standard in the us-central1 example |
| Microsoft Fabric native execution engine | Toggle in an environment or spark.native.enabled true; Fabric only |
| NVIDIA cuDF plugin for Apache Spark | Plugin enabled by a configuration setting and GPU instances; Spark 3.3.x to 4.2.0 across releases |
| Speedata | spark.plugins=com.speedata.spark.DashPlugin, spark.speedata.apus=1 and a PCIe card in a physical server or a Speedata-hosted AWS service |
Where DualBird's definition sits
DualBird says its plugin needs no code changes and that "the majority of common Spark performance configurations do not exist when using the solution". Its own engineer states that the FPGA path accelerates a defined set of operators (sort, join, aggregation and others) and that workloads dominated by Python UDFs or unsupported operators fall back to the CPU. Those are DualBird's statements, and DualBird does not publish a supported-operator list.
DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published.
Sources
- NVIDIA cuDF plugin for Apache Spark: FAQ, read 10 Nov 2026
- Databricks documentation: What is Photon, read 10 Nov 2026
- Google Cloud: Lightning Engine for Managed Service for Apache Spark, read 10 Nov 2026
- Google Cloud: Managed Service for Apache Spark pricing, read 10 Nov 2026
- Apache Spark documentation: Apache Arrow in PySpark, read 10 Nov 2026
- Apache Gluten: getting started, read 10 Nov 2026
- Apache Auron, read 10 Nov 2026
- Microsoft Learn: Native execution engine for Fabric Spark, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- DualBird: A simpler Spark, read 10 Nov 2026
- Flarion FAQs, read 10 Nov 2026
Last reviewed 10 Nov 2026.