Skip to content
Spark Acceleration Compare

Option profile, read 10 Nov 2026

Apache DataFusion Comet, Open-source Spark plugin on DataFusion: the published facts

Apache DataFusion Comet in the LedgerCompare with another option

Kind
Open-source Spark plugin on DataFusion
Website
datafusion.apache.org/comet
Name checked
Exists under the same name, Apache DataFusion Comet, as a subproject of Apache DataFusion (apache/datafusion-comet). No rename found.
Read
10 Nov 2026

What it does well

  • The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores.
  • Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03.

Where it is limited

  • The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported.
  • Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental.

Published speed results

FactSource typeSource, read
On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, 10 Nov 2026
The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, 10 Nov 2026

Code changes

FactSource typeSource, read
Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, 10 Nov 2026

Tuning

FactSource typeSource, read
The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, 10 Nov 2026

Spark versions, platforms and table formats

FactSource typeSource, read
Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, 10 Nov 2026
Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, 10 Nov 2026

Where it runs

FactSource typeSource, read
Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required.open source projectraw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, 10 Nov 2026

How open its benchmarks are

FactSource typeSource, read
The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page.open source projectdatafusion.apache.org/comet/contributor-guide/benchmarking.html, 10 Nov 2026

Company and breadth

FactSource typeSource, read
Licensed Apache-2.0 under the Apache Software Foundation; the docs site footer shows copyright 2023 to 2026 Apache Software Foundation. GitHub repo created 2024-01-15, about 1,300 stars.open source projectapi.github.com/repos/apache/datafusion-comet, 10 Nov 2026

Profile

FactSource typeSource, read
GitHub releases: 1.0.0 (2026-08-07), 1.1.0 (2026-10-06), 1.2.0 (2026-11-03), roughly monthly after 1.0.open source projectapi.github.com/repos/apache/datafusion-comet/releases?per_page=3, 10 Nov 2026

Ledger entries for Apache DataFusion Comet

PublishedOptionTestResult as publishedRan byMethod published
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository

Change Budget row

ColumnWhat changesDetailSource, read
CodeNo job code change statedComet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default.Comet compatibility guide, 10 Nov 2026
ClusterPlugin, shuffle manager, off-heap memoryEvery deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later.Comet tuning guide, 10 Nov 2026
CloudNone found on 2026-11-10No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances.Comet installation guide, 10 Nov 2026
VendorApache-2.0, no contract statedComet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described.Apache DataFusion Comet GitHub repository, 10 Nov 2026

Searched for and none found on 10 Nov 2026

  • Any published cost per job or cost-for-the-same-work result (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, datafusion.apache.org/comet/contributor-guide/benchmarking_aws_ec2.html)
  • Company sponsors or major corporate contributors (the fetched pages name only the Apache Software Foundation) (searched: datafusion.apache.org/comet/contributor-guide/benchmarking.html, api.github.com/repos/apache/datafusion-comet)

Apache DataFusion Comet side by side with each other option

Sources

  1. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, read 10 Nov 2026
  2. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  3. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  4. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  5. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  6. Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html, read 10 Nov 2026
  7. Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet, read 10 Nov 2026
  8. Apache DataFusion Comet: api.github.com/repos/apache/datafusion-comet/releases?per_page=3, read 10 Nov 2026

Last reviewed 10 Nov 2026.