Skip to content
Spark Acceleration Compare

Situation, read 10 Nov 2026

What to check when moving Spark from a managed platform to an open stack

All situationsOpen the Ledger

5 things to check, with the published facts that answer each

What does the bundled engine do that you would give up?

Check whether the engine you rely on exists outside its platform at all.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS documents the EMR runtime only where EMR, EMR on EKS and EMR Serverless are available.Amazon EMR pricing, 10 Nov 2026
Databricks PhotonDatabricks documents Photon only as part of Databricks compute on AWS, Azure and Google Cloud, not as a standalone engine for open source Spark or EMR.Databricks Photon documentation, 10 Nov 2026
Google Cloud Lightning EngineGoogle documents Lightning Engine on Google Cloud only.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineMicrosoft documents the native execution engine only inside Fabric, and it does not run on AWS EMR or EKS.Fabric native execution engine documentation, 10 Nov 2026

Which options run on your target platform?

Open stack here means Spark on EMR, EKS, Dataproc, Kubernetes or your own servers.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron builds for Spark 3.0 to 4.2 on Linux images and macOS. EMR, EKS or Databricks support: none found on 2026-11-10.Apache Auron build script, 10 Nov 2026
Apache DataFusion CometComet's jars bundle Linux native libraries for amd64 and arm64. The docs warn about cloud-provider Spark forks.Comet installation guide, 10 Nov 2026
Apache Gluten with VeloxGluten documents Ubuntu 20.04/22.04 and CentOS 8 as build and test platforms and has storage pages for S3, GCS and ABFS.Apache Gluten downloads, 10 Nov 2026
DataPelagoDataPelago documents EKS, Dataproc and GKE.DataPelago Google Cloud deployment docs, 10 Nov 2026
DualBirdDualBird documents Apache Spark, EMR and EKS on AWS F2 instances.DualBird home page, 10 Nov 2026
FlarionFlarion lists Databricks (init scripts), EMR (bootstrap action), Dataproc, HDInsight, Spark on Kubernetes and on-premises.Flarion product page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS, Dataproc 2.1 to 2.3 and EMR on EKS.NVIDIA cuDF plugin download page, 10 Nov 2026

What would your team operate that the platform used to?

Open-source plugins move memory sizing, jar loading and fallback checks to you.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronYou must build the Auron jar yourself (Rust nightly and a JDK), and the memory settings move off-heap into executor overhead.Apache Auron README, 10 Nov 2026
Apache DataFusion CometComet's tuning guide says every deployment must configure how much memory Comet can use, through the off-heap pool and executor memory overhead.Comet tuning guide, 10 Nov 2026
Apache Gluten with VeloxGluten needs off-heap memory enabled and sized, the columnar shuffle manager and binary jars loaded ahead of Spark's jars.Apache Gluten getting started, 10 Nov 2026
DataPelagoDataPelago's mandatory settings include the plugin class, subscription ID, license location, off-heap memory, the columnar shuffle manager and Netty options.DataPelago AWS deployment docs, 10 Nov 2026
FlarionFlarion says existing Spark tuning carries over, and shows each stage as native or Spark in telemetry.Flarion FAQ, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's guide covers executor and task counts, pinned memory, concurrent tasks per GPU and a procedure for GPU out-of-memory errors.NVIDIA Spark plugin user guide, 10 Nov 2026

Which license and contract applies?

An Apache-2.0 project and a metered subscription behave differently in procurement.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron is Apache-2.0 under the Apache Software Foundation.Apache Auron homepage, 10 Nov 2026
Apache DataFusion CometComet is Apache-2.0 under the Apache Software Foundation.Apache DataFusion Comet GitHub repository, 10 Nov 2026
Apache Gluten with VeloxGluten is Apache-2.0.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonPhoton is proprietary to Databricks and its source is not open.Photon paper, SIGMOD 2022, 10 Nov 2026
DataPelagoDataPelago's Nucleus and DPA-S are proprietary and licensed, billed pay-as-you-go through the AWS and GCP marketplaces.DataPelago home page, 10 Nov 2026
DualBirdDualBird publishes no price, price list or marketplace listing.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionFlarion is proprietary with usage-based list prices.Flarion pricing page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's source code is Apache-2.0 and its documentation CC-BY-4.0; you pay for the GPU instances.NVIDIA cuDF plugin license, 10 Nov 2026
SpeedataSpeedata sells servers or goes through OEM or cloud partners, with no price list published.Speedata product page, 10 Nov 2026

Which Spark versions move with you?

On a managed platform the Spark version is the platform's runtime. On an open stack you choose it, and each plugin supports a list.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2.Apache Auron build script, 10 Nov 2026
Apache DataFusion CometComet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later.Comet installation guide, 10 Nov 2026
Apache Gluten with VeloxGluten names Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported.Gluten Velox backend guide, 10 Nov 2026
Databricks PhotonDatabricks says the Spark version under Photon is the Databricks Runtime version, not a free choice of open source Spark.Databricks Photon documentation, 10 Nov 2026
DualBirdA DualBird list of supported Spark versions: none found on 2026-11-10.DualBird home page, 10 Nov 2026
FlarionFlarion publishes no supported Spark version list; its benchmark ran Spark 3.5.5.Flarion product page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkThe v26.10.0 release supports Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0.NVIDIA cuDF plugin download page, 10 Nov 2026

Sources

  1. Amazon EMR pricing, read 10 Nov 2026
  2. Databricks Photon documentation, read 10 Nov 2026
  3. Google Lightning Engine documentation, read 10 Nov 2026
  4. Fabric native execution engine documentation, read 10 Nov 2026
  5. Apache Auron build script, read 10 Nov 2026
  6. Comet installation guide, read 10 Nov 2026
  7. Apache Gluten downloads, read 10 Nov 2026
  8. DataPelago Google Cloud deployment docs, read 10 Nov 2026
  9. DualBird home page, read 10 Nov 2026
  10. Flarion product page, read 10 Nov 2026
  11. NVIDIA cuDF plugin download page, read 10 Nov 2026
  12. Apache Auron README, read 10 Nov 2026
  13. Comet tuning guide, read 10 Nov 2026
  14. Apache Gluten getting started, read 10 Nov 2026
  15. DataPelago AWS deployment docs, read 10 Nov 2026
  16. Flarion FAQ, read 10 Nov 2026
  17. NVIDIA Spark plugin user guide, read 10 Nov 2026
  18. Apache Auron homepage, read 10 Nov 2026
  19. Apache DataFusion Comet GitHub repository, read 10 Nov 2026
  20. Apache Gluten homepage, read 10 Nov 2026
  21. Photon paper, SIGMOD 2022, read 10 Nov 2026
  22. DataPelago home page, read 10 Nov 2026
  23. DualBird Iceberg compaction benchmark post, read 10 Nov 2026
  24. Flarion pricing page, read 10 Nov 2026
  25. NVIDIA cuDF plugin license, read 10 Nov 2026
  26. Speedata product page, read 10 Nov 2026
  27. Gluten Velox backend guide, read 10 Nov 2026

Last reviewed 10 Nov 2026.