# Apache DataFusion Comet, Open-source Spark plugin on DataFusion: the published facts

Option profile, read 10 Nov 2026

**The short answer.** Apache DataFusion Comet is an Apache-2.0 Spark plugin that runs supported operators on the Apache DataFusion native engine, written in Rust. The project's TPC-DS run at scale factor 1000 on an EKS cluster took 994 seconds on Spark 4.2.0 and 488 seconds on Comet 1.2.0. Release 1.2.0 shipped on 2026-11-03.

- **Kind:** Open-source Spark plugin on DataFusion
- **Website:** [datafusion.apache.org/comet](https://datafusion.apache.org/comet/)
- **Name checked:** Exists under the same name, Apache DataFusion Comet, as a subproject of Apache DataFusion (apache/datafusion-comet). No rename found.
- **Read:** 10 Nov 2026

## What it does well

- The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores.
- Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03.

## Where it is limited

- The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported.
- Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental.

## Published speed results

| Fact | Source type | Source, read |
| --- | --- | --- |
| On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 |
| The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 |

## Code changes

| Fact | Source type | Source, read |
| --- | --- | --- |
| Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 |

## Tuning

| Fact | Source type | Source, read |
| --- | --- | --- |
| The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md), 10 Nov 2026 |

## Spark versions, platforms and table formats

| Fact | Source type | Source, read |
| --- | --- | --- |
| Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 |
| Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 |

## Where it runs

| Fact | Source type | Source, read |
| --- | --- | --- |
| Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | open source project | [raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 |

## How open its benchmarks are

| Fact | Source type | Source, read |
| --- | --- | --- |
| The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | open source project | [datafusion.apache.org/comet/contributor-guide/benchmarking.html](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 |

## Company and breadth

| Fact | Source type | Source, read |
| --- | --- | --- |
| Licensed Apache-2.0 under the Apache Software Foundation; the docs site footer shows copyright 2023 to 2026 Apache Software Foundation. GitHub repo created 2024-01-15, about 1,300 stars. | open source project | [api.github.com/repos/apache/datafusion-comet](https://api.github.com/repos/apache/datafusion-comet), 10 Nov 2026 |

## Profile

| Fact | Source type | Source, read |
| --- | --- | --- |
| GitHub releases: 1.0.0 (2026-08-07), 1.1.0 (2026-10-06), 1.2.0 (2026-11-03), roughly monthly after 1.0. | open source project | [api.github.com/repos/apache/datafusion-comet/releases?per_page=3](https://api.github.com/repos/apache/datafusion-comet/releases?per_page=3), 10 Nov 2026 |

## Ledger entries for Apache DataFusion Comet

| Published | Option | Test | Result as published | Ran by | Method published |
| --- | --- | --- | --- | --- | --- |
| 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-11-03 | [Apache DataFusion Comet](/options/datafusion-comet/) | [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |

## Change Budget row

| Column | What changes | Detail | Source, read |
| --- | --- | --- | --- |
| Code | No job code change stated | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | [Comet compatibility guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md), 10 Nov 2026 |
| Cluster | Plugin, shuffle manager, off-heap memory | Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | [Comet tuning guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md), 10 Nov 2026 |
| Cloud | None found on 2026-11-10 | No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | [Comet installation guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md), 10 Nov 2026 |
| Vendor | Apache-2.0, no contract stated | Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | [Apache DataFusion Comet GitHub repository](https://api.github.com/repos/apache/datafusion-comet), 10 Nov 2026 |

## Searched for and none found on 10 Nov 2026

- Any published cost per job or cost-for-the-same-work result (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, datafusion.apache.org/comet/contributor-guide/benchmarking_aws_ec2.html)
- Company sponsors or major corporate contributors (the fetched pages name only the Apache Software Foundation) (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, api.github.com/repos/apache/datafusion-comet)

## Apache DataFusion Comet side by side with each other option

- [Apache DataFusion Comet and Amazon EMR runtime for Apache Spark](/compare/datafusion-comet-vs-emr-spark-runtime/)
- [Apache DataFusion Comet and Apache Auron](/compare/apache-auron-vs-datafusion-comet/)
- [Apache DataFusion Comet and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-datafusion-comet/)
- [Apache DataFusion Comet and Databricks Photon](/compare/databricks-photon-vs-datafusion-comet/)
- [Apache DataFusion Comet and DataPelago](/compare/datafusion-comet-vs-datapelago/)
- [Apache DataFusion Comet and DualBird](/compare/datafusion-comet-vs-dualbird/)
- [Apache DataFusion Comet and Flarion](/compare/datafusion-comet-vs-flarion/)
- [Apache DataFusion Comet and Google Cloud Lightning Engine](/compare/datafusion-comet-vs-dataproc-lightning-engine/)
- [Apache DataFusion Comet and Microsoft Fabric native execution engine](/compare/datafusion-comet-vs-fabric-native-execution-engine/)
- [Apache DataFusion Comet and NVIDIA cuDF plugin for Apache Spark](/compare/datafusion-comet-vs-nvidia-rapids-spark/)
- [Apache DataFusion Comet and Speedata](/compare/datafusion-comet-vs-speedata/)

## Sources

- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026)
- Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet/releases?per_page=3: https://api.github.com/repos/apache/datafusion-comet/releases?per_page=3 (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/datafusion-comet/
