Side by side, read 10 Nov 2026
Apache DataFusion Comet and Apache Gluten with Velox: the published facts in the same rows
Apache DataFusion Comet profileApache Gluten with Velox profile
The same rows for both
| Row | Apache DataFusion Comet | Apache Gluten with Velox |
|---|---|---|
| Kind | Open-source Spark plugin on DataFusion | Open-source Spark plugin offloading to Velox |
| Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. |
| Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. |
| Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. |
| Published cost results and prices | None found on 10 Nov 2026. | None found on 10 Nov 2026. |
| Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. |
| Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. |
| Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. |
| Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. |
| How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. |
| Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. |
| Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. |
| Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. |
| Ledger entries | Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 (run by its maker); Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 (run by its maker) | Gluten with Velox, TPC-DS 3 TB (project homepage) (run by its maker); Gluten with Velox, TPC-H 3 TB (project homepage) (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html, read 10 Nov 2026
- Apache Gluten with the Velox backend: gluten.apache.org, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: gluten.apache.org/downloads, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md, read 10 Nov 2026
- Apache DataFusion Comet GitHub repository, read 10 Nov 2026
- Apache Gluten Powered By page, read 10 Nov 2026
Last reviewed 10 Nov 2026.