# Apache DataFusion Comet and Google Cloud Lightning Engine: the published facts in the same rows

Side by side, read 10 Nov 2026

**The short answer.** Apache DataFusion Comet and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 published entries for Apache DataFusion Comet and 3 entries for Google Cloud Lightning Engine, each with who ran it.

## The same rows for both

| Row | Apache DataFusion Comet | Google Cloud Lightning Engine |
| --- | --- | --- |
| Kind | Open-source Spark plugin on DataFusion | Premium Spark engine on Google Cloud |
| Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. |
| Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. |
| Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. |
| Published cost results and prices | None found on 10 Nov 2026. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. |
| Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. |
| Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). |
| How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. |
| Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). |
| Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) |

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.

## How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded.

## Sources

- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)
- Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-dataproc-lightning-engine/
