Side by side, read 10 Nov 2026
Apache Auron and Apache DataFusion Comet: the published facts in the same rows
Apache Auron profileApache DataFusion Comet profile
The same rows for both
| Row | Apache Auron | Apache DataFusion Comet |
|---|---|---|
| Kind | Open-source Rust engine plugin for Spark | Open-source Spark plugin on DataFusion |
| Does well | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13. | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. |
| Limited | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. |
| Published speed results | The project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart). | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. |
| Published cost results and prices | None found on 10 Nov 2026. | None found on 10 Nov 2026. |
| Code changes | Auron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source. | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. |
| Tuning | The suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered. | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. |
| Spark versions, platforms and table formats | The build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages. | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. |
| Where it runs | Build needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found. | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. |
| How open its benchmarks are | The benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors). | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. |
| Change Budget: code | No job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes. | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. |
| Change Budget: cluster | Build the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB). | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10. | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. |
| Change Budget: vendor | Apache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. |
| Ledger entries | Auron TPC-DS 1 TB against Apache Spark 3.5.6 (run by its maker) | Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 (run by its maker); Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Apache Auron (incubating), formerly Blaze: auron.apache.org, read 10 Nov 2026
- Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html, read 10 Nov 2026
- Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh, read 10 Nov 2026
- Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html, read 10 Nov 2026
- Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html, read 10 Nov 2026
- Apache DataFusion Comet GitHub repository, read 10 Nov 2026
Last reviewed 10 Nov 2026.