Guide, read 10 Nov 2026
Operator coverage and fallback in Spark accelerators
Why coverage decides the result
An accelerator speeds up the operators it implements and hands the rest back to Spark. The published speedups come from benchmark queries that mostly use covered operators. Your jobs may use other ones. A job whose long stage contains one unsupported expression can see that stage run on the JVM, with format conversions on both sides. So the first technical question for any option is not how fast, but how much of my plan runs natively.
The sources below are the projects' own pages, read 2026-11-10. They differ in granularity, from whole-operator tables to lists of known differences. Where a project publishes no list, this guide says none found.
Where each project documents coverage
| Option (alphabetical) | Where coverage is documented | How to see fallback in a running job |
|---|---|---|
| Apache Auron | Homepage and docs describe fine-grained fallback for unsupported operators | Auron tab in the Spark UI records per-operator fallback reasons |
| Apache DataFusion Comet | Operator support page with a status per Spark physical operator: Supported, Supported (caveats), Planned; compatibility guide for expressions | Plan explain output marks fallback reasons with [COMET: ...] |
| Apache Gluten with Velox | Velox backend limitations page; per-format pages for Iceberg, Hudi and Paimon | Columnar-to-row and row-to-columnar conversion nodes in the plan; troubleshooting page |
| Databricks Photon | Photon documentation lists unsupported items: UDFs, RDD and Dataset APIs, stateful streaming | Spark UI plan shows which operators ran in Photon |
| DataPelago | Docs list limitations: ANSI mode and case-sensitive mode | Not stated on the pages read |
| DualBird | DualBird's engineer says a defined set (sort, join, aggregation and others); no operator list: none found on 2026-11-10 | Not stated: none found on 2026-11-10 |
| Flarion | FAQ says non-accelerated operations run on Spark | Telemetry marks each stage native or Spark with runtime and memory |
| Google Lightning Engine | Docs list unsupported inputs: Byte in ORC and Parquet; Struct, Array and Map in Parquet | Qualification tool checks jobs before migration |
| Microsoft Fabric native execution engine | Overview page lists differences on Runtime 1.3 | Fabric Spark Advisor alerts show fallback |
| NVIDIA cuDF plugin | Generated supported_ops page by operator and data type; compatibility page | spark.rapids.sql.explain=NOT_ON_GPU logs what stays on CPU |
| Speedata | Not published as a list on the pages read | Free Workload Analyzer predicts per-query acceleration from Spark logs |
Reading the lists
Comet's operator page says it runs whole subtrees of native operators together, so if a stage contains an operator Comet does not support, that stage falls back to regular Spark execution, and results are unaffected. Each operator can be switched off individually with spark.comet.exec.OPERATOR.enabled=false, which is a way to bisect a wrong result. Expressions whose native implementation differs from Spark are off unless allowIncompatible is set.
Gluten's Velox limitations page is a list of differences, not of operators: ANSI mode always falls back, regexp functions use RE2 with no lookaround, scans fully support only Parquet and partly ORC, and Hudi supports reading copy-on-write tables only. DataPelago, which configures the Gluten plugin, adds that native CPU acceleration is not used when ANSI mode is on. Both pages state that case-sensitive mode may give wrong results or is unsupported.
NVIDIA's supported-operators page is generated by a tool against Apache Spark 3.3.0, and the page itself says most of the content applies to other versions with variations. It lists expressions by data type. Decimal is a good example: Spark supports precision up to 38 digits, and the plugin supports 128-bit decimals from version 21.12. A job full of wide decimals needs a check on the type, not just the operator name.
What happens on fallback
The job still finishes. That is the documented behavior of every option above. What changes is cost. Comet and Gluten convert between columnar and row formats at each boundary. NVIDIA's FAQ adds that on fallback the job completes while the fallen-back stages are not accelerated. Photon falls back for the remainder of an operation. Lightning Engine's native query execution fails on GPU accelerators, per Google, which is a different failure mode from fallback.
No project publishes a measurement of the fallback penalty on a typical job: none found on 2026-11-10. You measure it in your own trial: compare the runtime of fully native stages with stages that fell back and note the conversion nodes in the plan.
A coverage check in four steps
1. Collect the plans. For your five longest jobs, capture the physical plan from the SQL tab of the Spark UI or from EXPLAIN FORMATTED, and list every operator and function in the long stages.
2. Look them up in the lists above. Mark each as covered, covered with caveats, or not listed. Pay attention to data types, ANSI mode and case sensitivity settings.
3. Run the vendor's own estimator on your logs where one exists: Flarion's free assessment (which Flarion says simulates each job from logs), Google's qualification tool, NVIDIA's Qualification tool or Speedata's Workload Analyzer. Treat their output as a prediction and check it in the trial.
4. Turn on the explain or UI view during the trial and count the stages that fall back. The trial guide puts this in the plan, and the guide on native engines explains the conversion boundary.
DualBird and coverage
DualBird says Spark still plans the job and its FPGA path covers a defined set of operators, with Python UDF-heavy work falling back to CPU, and that coverage is expanding. That is DualBird's own description. That description applies only on AWS F2 instances, not on Azure, Google Cloud or Databricks-managed clusters, and every DualBird benchmark published so far is DualBird's own.
Sources
- Apache DataFusion Comet: Spark operator support, read 10 Nov 2026
- Apache DataFusion Comet: compatibility guide, read 10 Nov 2026
- NVIDIA cuDF plugin: supported operators, read 10 Nov 2026
- NVIDIA cuDF plugin: Qualification tool, read 10 Nov 2026
- Apache Gluten: Velox backend limitations, read 10 Nov 2026
- Apache Auron, read 10 Nov 2026
- Databricks documentation: What is Photon, read 10 Nov 2026
- DataPelago: deploy on AWS, read 10 Nov 2026
- Google Cloud: Lightning Engine for Managed Service for Apache Spark, read 10 Nov 2026
- Microsoft Learn: Native execution engine for Fabric Spark, read 10 Nov 2026
- Flarion FAQs, read 10 Nov 2026
- Speedata Workload Analyzer, read 10 Nov 2026
- DualBird: A simpler Spark, read 10 Nov 2026
Last reviewed 10 Nov 2026.