# Apache DataFusion Comet and Flarion: the published facts in the same rows

Side by side, read 10 Nov 2026

**The short answer.** Apache DataFusion Comet (Open-source Spark plugin on DataFusion) and Flarion (Rust and DataFusion engine as one JAR) are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit, where each runs and how open the tests are. The Ledger holds 2 entries for Apache DataFusion Comet and 2 entries for Flarion.

## The same rows for both

| Row | Apache DataFusion Comet | Flarion |
| --- | --- | --- |
| Kind | Open-source Spark plugin on DataFusion | Rust and DataFusion engine as one JAR |
| Does well | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. Its benchmarks page publishes the machine, Spark 3.5.5 baseline, engine builds, memory budgets and per-query caps, and discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. |
| Limited | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. A published list of supported Spark versions: none found on 2026-11-10. Third-party benchmarks of Flarion: none found on 2026-11-10. |
| Published speed results | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. | The home page headline is 'Run faster, spend less, scale further, without changing a line of code', with tiles for 6 minute setup, 3x performance gains and 60% cost reduction. The tiles carry no workload or method on the page. The benchmarks page (September 2026) compares six engines on TPC-DS (103 queries) on one i4i.8xlarge reading Parquet from S3. Flarion is faster than vanilla Spark, Comet and Gluten, and slower than Databricks Photon in both scales. At 100 GB: Spark 1,135 s, Flarion 386 s, Photon 362 s. At 1 TB: Spark 135.6 min, Flarion 38.3 min, Photon 30.4 min. The page labels these as benchmark figures, not customer results, and not an audited TPC result. |
| Published cost results and prices | None found on 10 Nov 2026. | Usage-based list prices: self-hosted Spark on AWS, Azure or GCP $0.009 per CPU hour; fully managed Spark (EMR, GCP managed Spark, Azure HDInsight) $0.012 per CPU hour; Databricks $0.095 per DBU; on-premises $0.75 per CPU per month. Billing is on accelerated Spark cores (driver plus executors). A free assessment simulates each job from logs and reports projected runtime and compute. |
| Code changes | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. | Flarion is one JAR inside the existing cluster, with no agents or separate services. UDFs and UDAFs run on Spark as before. Operations Flarion does not accelerate run on Spark automatically. The page lists a 'one config change' install and says you can turn it off with one config change to compare. |
| Tuning | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. | The FAQ says existing Spark tuning carries over and is kept; Flarion changes how the engine executes. The garbage-collection post says native memory is managed off-heap with no GC pauses, but part of a job (driver, planning, unsupported operators, often the final write) still runs on the JVM heap. Telemetry marks each stage native or Spark with runtime and memory. |
| Spark versions, platforms and table formats | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. | Platforms listed: Databricks (init scripts), Amazon EMR (bootstrap action), Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes (Helm chart or Spark operator) and on-premises. Frameworks: Scala Spark, PySpark, Trino, Ray. Table formats: Iceberg, Delta Lake, Hudi. File formats: Parquet, Avro, JSON. The benchmark ran Spark 3.5.5 and Databricks 16.4 LTS (Spark 3.5.2). A supported-version list is not published; a blog post mentions staying on Spark 3.4 and gaining newer-Spark benefits. |
| Where it runs | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. | No special hardware: Flarion runs CPU-native on the fleet already in use (FAQ contrasts it with GPU accelerators). It is usable on AWS, Azure, GCP, on-premises and air-gapped setups. The company is SOC 2 Type II certified per the home page. |
| How open its benchmarks are | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. | The benchmark page publishes the machine, Spark 3.5.5 baseline, engine builds (Comet 1.0.0, Gluten 1.7.0 Velox, Databricks DBR 16.4 LTS), memory budgets, shuffle partitions (64), AQE on, no statistics/CBO, per-query caps (600 s at 100 GB, 1,800 s at 1 TB) and lower of two runs at 100 GB and one run at 1 TB. It discloses that q72 is counted at vanilla Spark's time for Gluten and Comet. It says the open-source engines can be rerun from public releases, but Flarion itself is not downloadable ('contact us'), and no repo or scripts were found. |
| Change Budget: code | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. | No code change stated. Flarion says UDFs and UDAFs run on Spark as before and operations it does not accelerate run on Spark automatically. The home page headline says 'without changing a line of code'. |
| Change Budget: cluster | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. | One JAR and one config change. Flarion is one JAR inside the existing cluster, installed with a one config change and switched off with one config change. Existing Spark tuning carries over. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. | No single cloud required. Flarion lists Databricks, Amazon EMR, Google Cloud Dataproc, Azure HDInsight, Spark on Kubernetes and on-premises, and says it supports air-gapped setups. |
| Change Budget: vendor | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. | Flarion license at usage-based prices. Flarion lists $0.009 per CPU hour (self-hosted), $0.012 (fully managed Spark), $0.095 per DBU (Databricks) and $0.75 per CPU per month (on premises), billed on accelerated Spark cores. |
| Ledger entries | [Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-1-3/) (run by its maker); [Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000](/ledger/datafusion-comet-tpcds-sf1000-vs-spark-4-2-0/) (run by its maker) | [Flarion TPC-DS SF100, six engines on one machine](/ledger/flarion-tpcds-sf100-six-engines/) (run by its maker); [Flarion TPC-DS SF1000, six engines on one machine](/ledger/flarion-tpcds-sf1000-six-engines/) (run by its maker) |

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.

## How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded.

## Sources

- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/compatibility/index.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/tuning.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/installation.md (read 10 Nov 2026)
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- Flarion: flarion.io: https://www.flarion.io/ (read 10 Nov 2026)
- Flarion: flarion.io/benchmarks: https://www.flarion.io/benchmarks (read 10 Nov 2026)
- Flarion: flarion.io/pricing: https://www.flarion.io/pricing (read 10 Nov 2026)
- Flarion: flarion.io/faqs: https://www.flarion.io/faqs (read 10 Nov 2026)
- Flarion: flarion.io/product: https://www.flarion.io/product (read 10 Nov 2026)
- Apache DataFusion Comet GitHub repository: https://api.github.com/repos/apache/datafusion-comet (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/datafusion-comet-vs-flarion/
