Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

Operator coverage and fallback in Spark accelerators

All guidesOpen the Ledger

Why coverage decides the result

An accelerator speeds up the operators it implements and hands the rest back to Spark. The published speedups come from benchmark queries that mostly use covered operators. Your jobs may use other ones. A job whose long stage contains one unsupported expression can see that stage run on the JVM, with format conversions on both sides. So the first technical question for any option is not how fast, but how much of my plan runs natively.

The sources below are the projects' own pages, read 2026-11-10. They differ in granularity, from whole-operator tables to lists of known differences. Where a project publishes no list, this guide says none found.

Where each project documents coverage

Read 2026-11-10 from each project's pages; see the sources below.
Option (alphabetical)Where coverage is documentedHow to see fallback in a running job
Apache AuronHomepage and docs describe fine-grained fallback for unsupported operatorsAuron tab in the Spark UI records per-operator fallback reasons
Apache DataFusion CometOperator support page with a status per Spark physical operator: Supported, Supported (caveats), Planned; compatibility guide for expressionsPlan explain output marks fallback reasons with [COMET: ...]
Apache Gluten with VeloxVelox backend limitations page; per-format pages for Iceberg, Hudi and PaimonColumnar-to-row and row-to-columnar conversion nodes in the plan; troubleshooting page
Databricks PhotonPhoton documentation lists unsupported items: UDFs, RDD and Dataset APIs, stateful streamingSpark UI plan shows which operators ran in Photon
DataPelagoDocs list limitations: ANSI mode and case-sensitive modeNot stated on the pages read
DualBirdDualBird's engineer says a defined set (sort, join, aggregation and others); no operator list: none found on 2026-11-10Not stated: none found on 2026-11-10
FlarionFAQ says non-accelerated operations run on SparkTelemetry marks each stage native or Spark with runtime and memory
Google Lightning EngineDocs list unsupported inputs: Byte in ORC and Parquet; Struct, Array and Map in ParquetQualification tool checks jobs before migration
Microsoft Fabric native execution engineOverview page lists differences on Runtime 1.3Fabric Spark Advisor alerts show fallback
NVIDIA cuDF pluginGenerated supported_ops page by operator and data type; compatibility pagespark.rapids.sql.explain=NOT_ON_GPU logs what stays on CPU
SpeedataNot published as a list on the pages readFree Workload Analyzer predicts per-query acceleration from Spark logs

Reading the lists

Comet's operator page says it runs whole subtrees of native operators together, so if a stage contains an operator Comet does not support, that stage falls back to regular Spark execution, and results are unaffected. Each operator can be switched off individually with spark.comet.exec.OPERATOR.enabled=false, which is a way to bisect a wrong result. Expressions whose native implementation differs from Spark are off unless allowIncompatible is set.

Gluten's Velox limitations page is a list of differences, not of operators: ANSI mode always falls back, regexp functions use RE2 with no lookaround, scans fully support only Parquet and partly ORC, and Hudi supports reading copy-on-write tables only. DataPelago, which configures the Gluten plugin, adds that native CPU acceleration is not used when ANSI mode is on. Both pages state that case-sensitive mode may give wrong results or is unsupported.

NVIDIA's supported-operators page is generated by a tool against Apache Spark 3.3.0, and the page itself says most of the content applies to other versions with variations. It lists expressions by data type. Decimal is a good example: Spark supports precision up to 38 digits, and the plugin supports 128-bit decimals from version 21.12. A job full of wide decimals needs a check on the type, not just the operator name.

What happens on fallback

The job still finishes. That is the documented behavior of every option above. What changes is cost. Comet and Gluten convert between columnar and row formats at each boundary. NVIDIA's FAQ adds that on fallback the job completes while the fallen-back stages are not accelerated. Photon falls back for the remainder of an operation. Lightning Engine's native query execution fails on GPU accelerators, per Google, which is a different failure mode from fallback.

No project publishes a measurement of the fallback penalty on a typical job: none found on 2026-11-10. You measure it in your own trial: compare the runtime of fully native stages with stages that fell back and note the conversion nodes in the plan.

A coverage check in four steps

1. Collect the plans. For your five longest jobs, capture the physical plan from the SQL tab of the Spark UI or from EXPLAIN FORMATTED, and list every operator and function in the long stages.

2. Look them up in the lists above. Mark each as covered, covered with caveats, or not listed. Pay attention to data types, ANSI mode and case sensitivity settings.

3. Run the vendor's own estimator on your logs where one exists: Flarion's free assessment (which Flarion says simulates each job from logs), Google's qualification tool, NVIDIA's Qualification tool or Speedata's Workload Analyzer. Treat their output as a prediction and check it in the trial.

4. Turn on the explain or UI view during the trial and count the stages that fall back. The trial guide puts this in the plan, and the guide on native engines explains the conversion boundary.

DualBird and coverage

DualBird says Spark still plans the job and its FPGA path covers a defined set of operators, with Python UDF-heavy work falling back to CPU, and that coverage is expanding. That is DualBird's own description. That description applies only on AWS F2 instances, not on Azure, Google Cloud or Databricks-managed clusters, and every DualBird benchmark published so far is DualBird's own.

Sources

  1. Apache DataFusion Comet: Spark operator support, read 10 Nov 2026
  2. Apache DataFusion Comet: compatibility guide, read 10 Nov 2026
  3. NVIDIA cuDF plugin: supported operators, read 10 Nov 2026
  4. NVIDIA cuDF plugin: Qualification tool, read 10 Nov 2026
  5. Apache Gluten: Velox backend limitations, read 10 Nov 2026
  6. Apache Auron, read 10 Nov 2026
  7. Databricks documentation: What is Photon, read 10 Nov 2026
  8. DataPelago: deploy on AWS, read 10 Nov 2026
  9. Google Cloud: Lightning Engine for Managed Service for Apache Spark, read 10 Nov 2026
  10. Microsoft Learn: Native execution engine for Fabric Spark, read 10 Nov 2026
  11. Flarion FAQs, read 10 Nov 2026
  12. Speedata Workload Analyzer, read 10 Nov 2026
  13. DualBird: A simpler Spark, read 10 Nov 2026

Last reviewed 10 Nov 2026.