Situation, read 10 Nov 2026
What to check when moving Spark from a managed platform to an open stack
5 things to check, with the published facts that answer each
What does the bundled engine do that you would give up?
Check whether the engine you rely on exists outside its platform at all.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS documents the EMR runtime only where EMR, EMR on EKS and EMR Serverless are available. | Amazon EMR pricing, 10 Nov 2026 |
| Databricks Photon | Databricks documents Photon only as part of Databricks compute on AWS, Azure and Google Cloud, not as a standalone engine for open source Spark or EMR. | Databricks Photon documentation, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google documents Lightning Engine on Google Cloud only. | Google Lightning Engine documentation, 10 Nov 2026 |
| Microsoft Fabric native execution engine | Microsoft documents the native execution engine only inside Fabric, and it does not run on AWS EMR or EKS. | Fabric native execution engine documentation, 10 Nov 2026 |
Which options run on your target platform?
Open stack here means Spark on EMR, EKS, Dataproc, Kubernetes or your own servers.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron builds for Spark 3.0 to 4.2 on Linux images and macOS. EMR, EKS or Databricks support: none found on 2026-11-10. | Apache Auron build script, 10 Nov 2026 |
| Apache DataFusion Comet | Comet's jars bundle Linux native libraries for amd64 and arm64. The docs warn about cloud-provider Spark forks. | Comet installation guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten documents Ubuntu 20.04/22.04 and CentOS 8 as build and test platforms and has storage pages for S3, GCS and ABFS. | Apache Gluten downloads, 10 Nov 2026 |
| DataPelago | DataPelago documents EKS, Dataproc and GKE. | DataPelago Google Cloud deployment docs, 10 Nov 2026 |
| DualBird | DualBird documents Apache Spark, EMR and EKS on AWS F2 instances. | DualBird home page, 10 Nov 2026 |
| Flarion | Flarion lists Databricks (init scripts), EMR (bootstrap action), Dataproc, HDInsight, Spark on Kubernetes and on-premises. | Flarion product page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS, Dataproc 2.1 to 2.3 and EMR on EKS. | NVIDIA cuDF plugin download page, 10 Nov 2026 |
What would your team operate that the platform used to?
Open-source plugins move memory sizing, jar loading and fallback checks to you.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | You must build the Auron jar yourself (Rust nightly and a JDK), and the memory settings move off-heap into executor overhead. | Apache Auron README, 10 Nov 2026 |
| Apache DataFusion Comet | Comet's tuning guide says every deployment must configure how much memory Comet can use, through the off-heap pool and executor memory overhead. | Comet tuning guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten needs off-heap memory enabled and sized, the columnar shuffle manager and binary jars loaded ahead of Spark's jars. | Apache Gluten getting started, 10 Nov 2026 |
| DataPelago | DataPelago's mandatory settings include the plugin class, subscription ID, license location, off-heap memory, the columnar shuffle manager and Netty options. | DataPelago AWS deployment docs, 10 Nov 2026 |
| Flarion | Flarion says existing Spark tuning carries over, and shows each stage as native or Spark in telemetry. | Flarion FAQ, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA's guide covers executor and task counts, pinned memory, concurrent tasks per GPU and a procedure for GPU out-of-memory errors. | NVIDIA Spark plugin user guide, 10 Nov 2026 |
Which license and contract applies?
An Apache-2.0 project and a metered subscription behave differently in procurement.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron is Apache-2.0 under the Apache Software Foundation. | Apache Auron homepage, 10 Nov 2026 |
| Apache DataFusion Comet | Comet is Apache-2.0 under the Apache Software Foundation. | Apache DataFusion Comet GitHub repository, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten is Apache-2.0. | Apache Gluten homepage, 10 Nov 2026 |
| Databricks Photon | Photon is proprietary to Databricks and its source is not open. | Photon paper, SIGMOD 2022, 10 Nov 2026 |
| DataPelago | DataPelago's Nucleus and DPA-S are proprietary and licensed, billed pay-as-you-go through the AWS and GCP marketplaces. | DataPelago home page, 10 Nov 2026 |
| DualBird | DualBird publishes no price, price list or marketplace listing. | DualBird Iceberg compaction benchmark post, 10 Nov 2026 |
| Flarion | Flarion is proprietary with usage-based list prices. | Flarion pricing page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA's source code is Apache-2.0 and its documentation CC-BY-4.0; you pay for the GPU instances. | NVIDIA cuDF plugin license, 10 Nov 2026 |
| Speedata | Speedata sells servers or goes through OEM or cloud partners, with no price list published. | Speedata product page, 10 Nov 2026 |
Which Spark versions move with you?
On a managed platform the Spark version is the platform's runtime. On an open stack you choose it, and each plugin supports a list.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2. | Apache Auron build script, 10 Nov 2026 |
| Apache DataFusion Comet | Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Comet installation guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten names Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported. | Gluten Velox backend guide, 10 Nov 2026 |
| Databricks Photon | Databricks says the Spark version under Photon is the Databricks Runtime version, not a free choice of open source Spark. | Databricks Photon documentation, 10 Nov 2026 |
| DualBird | A DualBird list of supported Spark versions: none found on 2026-11-10. | DualBird home page, 10 Nov 2026 |
| Flarion | Flarion publishes no supported Spark version list; its benchmark ran Spark 3.5.5. | Flarion product page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | The v26.10.0 release supports Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0. | NVIDIA cuDF plugin download page, 10 Nov 2026 |
Sources
- Amazon EMR pricing, read 10 Nov 2026
- Databricks Photon documentation, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- Fabric native execution engine documentation, read 10 Nov 2026
- Apache Auron build script, read 10 Nov 2026
- Comet installation guide, read 10 Nov 2026
- Apache Gluten downloads, read 10 Nov 2026
- DataPelago Google Cloud deployment docs, read 10 Nov 2026
- DualBird home page, read 10 Nov 2026
- Flarion product page, read 10 Nov 2026
- NVIDIA cuDF plugin download page, read 10 Nov 2026
- Apache Auron README, read 10 Nov 2026
- Comet tuning guide, read 10 Nov 2026
- Apache Gluten getting started, read 10 Nov 2026
- DataPelago AWS deployment docs, read 10 Nov 2026
- Flarion FAQ, read 10 Nov 2026
- NVIDIA Spark plugin user guide, read 10 Nov 2026
- Apache Auron homepage, read 10 Nov 2026
- Apache DataFusion Comet GitHub repository, read 10 Nov 2026
- Apache Gluten homepage, read 10 Nov 2026
- Photon paper, SIGMOD 2022, read 10 Nov 2026
- DataPelago home page, read 10 Nov 2026
- DualBird Iceberg compaction benchmark post, read 10 Nov 2026
- Flarion pricing page, read 10 Nov 2026
- NVIDIA cuDF plugin license, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- Gluten Velox backend guide, read 10 Nov 2026
Last reviewed 10 Nov 2026.