Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Databricks Photon and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows

Databricks Photon profileNVIDIA cuDF plugin for Apache Spark profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowDatabricks PhotonNVIDIA cuDF plugin for Apache Spark
KindDatabricks's C++ vectorized engineGPU plugin, formerly RAPIDS Accelerator
Does wellPer Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021.AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported.
LimitedPhoton is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not.
Published speed resultsDocs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job.AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks.
Published cost results and pricesPhoton instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing.AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances.
Code changesDocs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus.
TuningPhoton is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback.The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU.
Spark versions, platforms and table formatsPhoton accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark.Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables.
Where it runsPhoton is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses.Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs.
How open its benchmarks areThe TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found.NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts.
Change Budget: codeNo change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU.
Change Budget: clusterSet runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate.GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager.
Change Budget: cloudDatabricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud.AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0.
Change Budget: vendorDatabricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page.Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges.
Ledger entriesPhoton customer speedups, SIGMOD 2022 paper (run by its maker); Databricks audited 100 TB TPC-DS result on Photon (run by its maker)GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB (run by its maker); NVIDIA comparison with open-source Spark on EC2, NDS 3 TB (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. Databricks Photon: docs.databricks.com/en/compute/photon.html, read 10 Nov 2026
  2. Databricks Photon: databricks.com/product/pricing, read 10 Nov 2026
  3. Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf, read 10 Nov 2026
  4. Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing, read 10 Nov 2026
  5. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e, read 10 Nov 2026
  6. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html, read 10 Nov 2026
  7. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html, read 10 Nov 2026
  8. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html, read 10 Nov 2026
  9. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html, read 10 Nov 2026
  10. NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md, read 10 Nov 2026

Last reviewed 10 Nov 2026.