# Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark: the published facts in the same rows

Side by side, read 10 Nov 2026

**The short answer.** Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for NVIDIA cuDF plugin for Apache Spark.

## The same rows for both

| Row | Apache DataFusion Comet | NVIDIA cuDF plugin for Apache Spark |
| --- | --- | --- |
| Kind | Open-source Spark plugin on DataFusion | GPU plugin, formerly RAPIDS Accelerator |
| Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. Amazon EMR on EKS supports the plugin natively from release 6.9.0, and NVIDIA lists Databricks 14.3 ML LTS and 17.3 ML LTS and Dataproc 2.1 to 2.3 as supported. |
| Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. Only the DataFrame API and a limited set of operations are accelerated; RDDs, GraphX and APIs that take Scala or Java functions are not. |
| Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | AWS and NVIDIA report that TPC-DS at 3 TB (103 queries) on 8-node Amazon EMR on EKS clusters ran in 281 seconds on g7.4xlarge GPU instances (NVIDIA RTX PRO 4500 Blackwell) versus 1,010 to 1,043 seconds on comparable CPU instances, which AWS labels a 3.7x speedup. Median of 5 iterations, Spark 3.5.6, cuDF plugin 26.04.2, us-east-1. NVIDIA's own FAQ gives a general expectation and says there is no formula for a specific query. This is a vendor claim with no named benchmark behind it. NVIDIA's overview page compares the published AWS GPU result with separate NDS (TPC-DS derived) 3 TB runs of open-source Spark 3.5.6 on EC2. It says the GPU run is 8.4x faster than m7gd.8xlarge and 11.0x faster than m9gd.4xlarge, with 77% and 72% lower reported cost per run. NVIDIA itself notes the runs use different hardware and software stacks. |
| Published cost results and prices | None found on 10 Nov 2026. | AWS reports cost per benchmark run of $2.06 on the GPU cluster versus $2.93 to $3.18 on the CPU clusters, using On-Demand us-east-1 prices as of May 2026. The g7.4xlarge cluster costs $26.35 per hour (8 x $3.042 EC2 plus $2.01 for EMR on EKS) against $10.43 to $10.99 per hour for the CPU clusters. Cost excludes the EKS control-plane fee, EBS, S3 and the driver pod. The software itself is free; you pay for GPU instances. |
| Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU, so the job completes but those stages are not accelerated. Only the DataFrame API (Dataset[Row]) and a limited set of SQL and DataFrame operations are accelerated. Custom classes or types with Dataset, APIs that take Row or Scala/Java functions (flatMap, foreach, foreachPartition), GraphX and RDDs are not accelerated. Structured streaming is not a focus. |
| Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | The user guide has a tuning guide (executors, tasks per executor, pooled and pinned memory, concurrent tasks per GPU, shuffle partitions, input partition size, columnar batch size), an AutoTuner, a Profiling tool and a Qualification tool, plus a recommended-practices section on handling GPU out-of-memory errors. In the AWS test the GPU config set executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager, and AWS advises using the Spark History Server to tune concurrentGpuTasks and pinnedPool.size. AWS recommends a pre-migration assessment with the NVIDIA qualification tool to estimate GPU acceleration potential, and setting spark.rapids.sql.explain=NOT_ON_GPU to see which operations run on CPU. |
| Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Release v26.10.0 supports Apache Spark 3.3.x, 3.4.x, 3.5.0 to 3.5.9, 4.0.0 to 4.0.4, 4.1.1 to 4.1.3 and 4.2.0 (Scala 2.12 and 2.13 depending on version). Supported Databricks runtimes on AWS and Azure are 14.3 ML LTS (Spark 3.5.0) and 17.3 ML LTS (Spark 4.0.0). Dataproc 2.1 to 2.3 and Dataproc Serverless runtimes 1.2, 2.2 and 3.0 are supported. Amazon EMR on EKS natively supports the plugin (release 6.9.0 or later; the '-spark-rapids' release variant ships it pre-installed). The user guide also has a getting-started page for AWS EMR (EC2) and covers Apache Iceberg and Delta Lake support pages, including reading and writing tables. |
| Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Needs NVIDIA GPUs: Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generations; tested on V100, T4, A10, A100, L4, H100, B100, RTX PRO 4500 and RTX PRO 6000. Linux with glibc 2.28 or newer; NVIDIA driver R525+ (CUDA 12) or R580+ (CUDA 13). On AWS the benchmark used g7.4xlarge (1 GPU, 16 vCPU, 64 GB RAM); NVIDIA's EMR guide maps m5d.8xlarge to g4dn.8xlarge for T4 GPUs. |
| How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | NVIDIA publishes the NDS benchmark automation (data generation, power and throughput runs) in a public Apache-2.0 repo, with a disclaimer that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. The AWS post publishes the cluster, Spark config, EMR release and pricing method; it does not link scripts. |
| Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. NVIDIA says existing Spark applications run with no code change once the plugin is launched and enabled by a configuration setting. Unsupported operators and UDFs fall back to the CPU. |
| Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | GPU instances and plugin settings. The cluster needs NVIDIA GPUs (Volta to Blackwell generations) and a current driver. AWS's test used g7.4xlarge nodes with executor memoryOverhead 30G, pinnedPool 8G, concurrentGpuTasks 3 and RapidsShuffleManager. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | AWS, Azure and Google Cloud documented. NVIDIA lists Databricks runtimes on AWS and Azure and Dataproc on Google Cloud; Amazon EMR on EKS supports the plugin natively from release 6.9.0. |
| Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Free software; GPU instances paid. The software is free (source Apache-2.0); you pay for GPU instances, and AWS's cost per run includes EC2 and EMR on EKS charges. |
| Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB](/ledger/nvidia-cudf-plugin-emr-on-eks-tpcds-3tb/) (run by its maker); [NVIDIA comparison with open-source Spark on EC2, NDS 3 TB](/ledger/nvidia-cudf-plugin-nds-3tb-vs-ec2-spark/) (run by its maker) |

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.

## How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded.

## Sources

- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-e: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/faq.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/overview.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/overview.html (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): docs.nvidia.com/spark-rapids/user-guide/latest/index.html: https://docs.nvidia.com/spark-rapids/user-guide/latest/index.html (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): nvidia.github.io/cudf-spark/docs/download.html: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026)
- NVIDIA RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF plugin for Apache Spark): raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md: https://raw.githubusercontent.com/NVIDIA/cudf-spark-benchmarks/dev/nds/README.md (read 10 Nov 2026)
- Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-nvidia-rapids-spark/
