# Apache DataFusion Comet and Microsoft Fabric native execution engine: the published facts in the same rows

Side by side, read 10 Nov 2026

**The short answer.** Apache DataFusion Comet and Microsoft Fabric native execution engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 2 entries for Apache DataFusion Comet and no entries for Microsoft Fabric native execution engine.

## The same rows for both

| Row | Apache DataFusion Comet | Microsoft Fabric native execution engine |
| --- | --- | --- |
| Kind | Open-source Spark plugin on DataFusion | Velox-based engine inside Fabric Spark |
| Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. The engine works with Parquet, Delta and CSV, including data reached through OneLake shortcuts. |
| Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. Microsoft's page gives no number, workload or dataset for the speedup, and a benchmark method or scripts: none found on 2026-11-10. |
| Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The page says the engine significantly raises query performance while minimizing operational costs, but gives no number, workload or dataset. 'Actual results vary by workload characteristics and configuration.' |
| Published cost results and prices | None found on 10 Nov 2026. | Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. An opt-in Autoscale Billing for Spark runs Spark jobs on dedicated serverless resources at standard regional pay-as-you-go CU-hour rates, with a base Fabric capacity still required. SKU prices on the page load dynamically and were not readable. |
| Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Docs say no code modifications are required and no vendor lock-in. Unsupported operators fall back to JVM Spark. Listed limitations include decimal-to-float casting, timezone, round(), map() duplicate-key and collect_list ordering differences on Runtime 1.3, said to be resolved in Runtime 2.0. |
| Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | Enable via the Acceleration tab toggle in an environment, or set spark.native.enabled to true at the start of a notebook or Spark job definition. Per-query disable is by setting the flag to false and re-enabling in later cells. Fabric Spark Advisor alerts show fallback. |
| Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Supports Runtime 1.3 (Spark 3.5, Delta Lake 3.2) and Runtime 2.0 (Spark 4.1, Delta Lake 4.1). Works with Parquet, Delta and CSV, including data reached via OneLake shortcuts. Not an AWS service. |
| Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Runs only inside Microsoft Fabric (Azure) on OneLake data. It does not run on AWS EMR or EKS. |
| How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | No benchmark method, data or scripts are published for the engine's speedup on the documentation page. |
| Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code modifications stated. Microsoft's documentation says no code modifications are required. Unsupported operators fall back to JVM Spark. |
| Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Toggle in environment or Spark config. Enable it with the Acceleration tab toggle in an environment, or set spark.native.enabled to true in a notebook or Spark job definition. It needs Runtime 1.3 or 2.0. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Microsoft Fabric on Azure only. The engine runs only inside Microsoft Fabric on OneLake data, and does not run on AWS EMR or EKS. |
| Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Fabric capacity required. Fabric is billed by capacity units (F2 to F4096 SKUs), pay-as-you-go or reservation. Autoscale Billing for Spark still requires a base Fabric capacity. SKU prices were not readable on the page. |
| Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | None found on 10 Nov 2026. |

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.

## How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded.

## Sources

- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- Microsoft Fabric Spark native execution engine: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026)
- Microsoft Fabric Spark native execution engine: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026)
- Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-fabric-native-execution-engine/
