Situation, read 10 Nov 2026
What to check when nightly Spark pipelines no longer finish before morning
5 things to check, with the published facts that answer each
Which runtimes have been published, on what workload and size?
Compare seconds and minutes only within one test. Each test below used its own cluster and baseline.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS says up to 4.5x faster than open-source equivalents, with no workload or scale stated. | Amazon EMR Spark features page, 10 Nov 2026 |
| Apache Auron | The Auron project says about 2x faster than Spark 3.5 on TPC-DS 1 TB (Spark 3.5.6 against Auron 6.0.0-preview on YARN); per-query numbers are a chart image only. | Apache Auron benchmark page, 10 Nov 2026 |
| Apache DataFusion Comet | TPC-DS derived, scale factor 1000, EKS on r6i: Spark 4.2.0 994 s, Comet 1.1.0 601 s, Comet 1.2.0 488 s. | Comet TPC-DS benchmark results, 10 Nov 2026 |
| Apache Gluten with Velox | The Gluten homepage reports 3.34x on TPC-H and 3.02x on TPC-DS at 3 TB with Spark 3.3.1 on Intel hardware, with no method linked. | Apache Gluten homepage, 10 Nov 2026 |
| Databricks Photon | The SIGMOD 2022 paper reports an average customer speedup of 3x over the previous Databricks Runtime, with a maximum over 10x. | Photon paper, SIGMOD 2022, 10 Nov 2026 |
| DataPelago | DataPelago publishes operator microbenchmarks against NVIDIA cuDF (up to 10.5x on project, 8.2x on K-largest selection). An end-to-end Spark benchmark: none found on 2026-11-10. | DataPelago Nucleus versus cuDF post, 10 Nov 2026 |
| DualBird | DualBird's June 2026 post says about 12x to 20x faster Spark task execution on a 100 GB Iceberg compaction; its home page shows 6x to 18x on a 110 GB enrichment pipeline with no method. | DualBird Iceberg compaction benchmark post, 10 Nov 2026 |
| Flarion | Flarion, TPC-DS SF1000 on one i4i.8xlarge: Spark 3.5.5 135.6 min, Flarion 38.3 min, Photon 30.4 min. | Flarion benchmarks page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google reports 3.6x on TPC-H-like queries at 10 TB and, on its pricing page, up to 4.9x with no workload stated. | Google Lightning Engine launch blog, 10 Nov 2026 |
| Microsoft Fabric native execution engine | A benchmark number for the Fabric native execution engine: none found on 2026-11-10. | Fabric native execution engine documentation, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | TPC-DS 3 TB on 8 nodes of EMR on EKS: GPU 281 s, CPU clusters 1,010 to 1,043 s (AWS and NVIDIA). | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata's home page shows 62.7x on production Spark workloads, with no customer, data size or method. | Speedata home page, 10 Nov 2026 |
Which parts of your jobs stay on the old path?
An accelerator only helps the operators it takes over. Jobs heavy in UDFs or RDDs spend most of their time on the path that did not change.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Gluten with Velox | Gluten falls back through columnar-to-row conversions; the docs list ANSI mode as always falling back and Parquet as the only fully supported scan format. | Gluten Velox backend limitations, 10 Nov 2026 |
| Databricks Photon | Databricks says Photon does not support UDFs, RDD APIs or Dataset APIs, and queries under about two seconds see no meaningful gain. | Databricks Photon documentation, 10 Nov 2026 |
| DataPelago | DataPelago tells users to avoid heavy use of RDDs and UDFs, and lists ANSI mode as running on the JVM. | DataPelago Accelerator for Spark page, 10 Nov 2026 |
| DualBird | A DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | DualBird post, A Simpler Spark, 10 Nov 2026 |
| Flarion | Flarion says UDFs and UDAFs run on Spark as before, and the driver, planning, unsupported operators and often the final write still run on the JVM heap. | Flarion FAQ, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google says Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. | Google Lightning Engine documentation, 10 Nov 2026 |
| Microsoft Fabric native execution engine | Microsoft says unsupported operators fall back to JVM Spark, and lists casting, timezone, round() and collect_list ordering differences on Runtime 1.3. | Fabric native execution engine documentation, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA says RDDs, GraphX and APIs that take Scala or Java functions are not accelerated, and unsupported operators fall back to the CPU. | NVIDIA Spark plugin FAQ, 10 Nov 2026 |
| Speedata | Speedata says SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. | Speedata product page, 10 Nov 2026 |
Can you predict the gain before you change the cluster?
A pre-check on your own logs separates a TPC-DS result from your pipelines.
| Option | Published fact | Source, read |
|---|---|---|
| Apache DataFusion Comet | Comet marks fallback reasons in plan explain output, which shows what stays on Spark. | Comet compatibility guide, 10 Nov 2026 |
| Databricks Photon | Databricks tells users to check the Spark UI to see Photon fallback. | Databricks Photon documentation, 10 Nov 2026 |
| Flarion | Flarion offers a free assessment that simulates each job from logs and reports projected runtime and compute. | Flarion pricing page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google documents a qualification tool for checking jobs. | Google Lightning Engine documentation, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA documents a Qualification tool, a Profiling tool and an AutoTuner in its user guide. | NVIDIA Spark plugin user guide, 10 Nov 2026 |
| Speedata | Speedata's Workload Analyzer predicts per-query acceleration before purchase from uploaded Spark logs. | Speedata Workload Analyzer page, 10 Nov 2026 |
Is the slow part compute, or shuffle, spill and I/O?
Native engines speed up compute. A pipeline waiting on shuffle, small files or object storage may not move much.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron supports Celeborn 0.5 and 0.6 and Uniffle 0.10 as remote shuffle services. | Apache Auron build script, 10 Nov 2026 |
| Apache DataFusion Comet | Comet supports remote shuffle through Celeborn. | Comet Iceberg guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten supports Celeborn 0.3 to 0.5 and Uniffle 0.9.2 for remote shuffle. | Gluten Velox Iceberg page, 10 Nov 2026 |
| Databricks Photon | Databricks says queries under about two seconds see no meaningful gain from Photon. | Databricks Photon documentation, 10 Nov 2026 |
| DualBird | DualBird says tens of Spark settings become irrelevant, spills effectively disappear and skew stops causing spills. A later post says skew still costs time, but less. These are DualBird's claims. | DualBird product page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google says I/O-bound or RDD-heavy jobs may suit the standard engine. | Google Lightning Engine documentation, 10 Nov 2026 |
Can you rerun the benchmark yourself?
A method that can be repeated on your own data can be checked; a number that can only be read cannot.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS publishes aws-samples/emr-spark-benchmark for a 3 TB TPC-DS run on OSS Spark and EMR; the repository states no results. | aws-samples emr-spark-benchmark repository, 10 Nov 2026 |
| Apache Auron | The Auron benchmark page links its data generator and runner tools and the full Spark configuration. | Apache Auron benchmark page, 10 Nov 2026 |
| Apache DataFusion Comet | Comet publishes benchmark scripts, a data generation repository and its non-default Spark settings. | Comet benchmarking guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten ships the gluten-it tool for TPC-style runs; the homepage numbers do not link a run configuration. | Gluten integration test tool README, 10 Nov 2026 |
| DualBird | DualBird's white paper has a landing page, and the full document is behind a form. No public repository was found. | DualBird compaction white paper page, 10 Nov 2026 |
| Flarion | Flarion publishes its machine, builds, settings and caps, says the open-source engines can be rerun from public releases, and does not offer Flarion for download. | Flarion benchmarks page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA publishes NDS benchmark automation in a public Apache-2.0 repository and notes NDS is not comparable to published TPC-DS results. | NVIDIA NDS benchmark README, 10 Nov 2026 |
| Speedata | Speedata's customer tiles have no published method, and its white paper is behind a form. | Speedata GitHub organization, 10 Nov 2026 |
Sources
- Amazon EMR Spark features page, read 10 Nov 2026
- Apache Auron benchmark page, read 10 Nov 2026
- Comet TPC-DS benchmark results, read 10 Nov 2026
- Apache Gluten homepage, read 10 Nov 2026
- Photon paper, SIGMOD 2022, read 10 Nov 2026
- DataPelago Nucleus versus cuDF post, read 10 Nov 2026
- DualBird Iceberg compaction benchmark post, read 10 Nov 2026
- Flarion benchmarks page, read 10 Nov 2026
- Google Lightning Engine launch blog, read 10 Nov 2026
- Fabric native execution engine documentation, read 10 Nov 2026
- AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
- Speedata home page, read 10 Nov 2026
- Gluten Velox backend limitations, read 10 Nov 2026
- Databricks Photon documentation, read 10 Nov 2026
- DataPelago Accelerator for Spark page, read 10 Nov 2026
- DualBird post, A Simpler Spark, read 10 Nov 2026
- Flarion FAQ, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- NVIDIA Spark plugin FAQ, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- Comet compatibility guide, read 10 Nov 2026
- Flarion pricing page, read 10 Nov 2026
- NVIDIA Spark plugin user guide, read 10 Nov 2026
- Speedata Workload Analyzer page, read 10 Nov 2026
- Apache Auron build script, read 10 Nov 2026
- Comet Iceberg guide, read 10 Nov 2026
- Gluten Velox Iceberg page, read 10 Nov 2026
- DualBird product page, read 10 Nov 2026
- aws-samples emr-spark-benchmark repository, read 10 Nov 2026
- Comet benchmarking guide, read 10 Nov 2026
- Gluten integration test tool README, read 10 Nov 2026
- DualBird compaction white paper page, read 10 Nov 2026
- NVIDIA NDS benchmark README, read 10 Nov 2026
- Speedata GitHub organization, read 10 Nov 2026
Last reviewed 10 Nov 2026.