Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Apache DataFusion Comet and Speedata: the published facts in the same rows

Apache DataFusion Comet profileSpeedata profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowApache DataFusion CometSpeedata
KindOpen-source Spark plugin on DataFusionPCIe analytics chip plus Dash Spark plugin
Does wellThe project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03.Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase.
LimitedThe docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental.The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read.
Published speed resultsOn an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s.The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details.
Published cost results and pricesNone found on 10 Nov 2026.No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote.
Code changesComet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default.Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1.
TuningThe tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead.The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase.
Spark versions, platforms and table formatsComet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported.The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned.
Where it runsPublished Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required.The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026.
How open its benchmarks areThe project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page.The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read.
Change Budget: codeNo job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default.No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU.
Change Budget: clusterPlugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later.PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1.
Change Budget: cloudNone found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances.On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3.
Change Budget: vendorApache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described.Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners.
Ledger entriesComet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 (run by its maker); Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 (run by its maker)Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper (run by Microsoft (paper); numbers read from Speedata's blog); Speedata production Spark workload tile (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, read 10 Nov 2026
  2. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  3. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  4. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  5. Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
  6. Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html, read 10 Nov 2026
  7. Speedata: speedata.io, read 10 Nov 2026
  8. Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-, read 10 Nov 2026
  9. Speedata: speedata.io/product, read 10 Nov 2026
  10. Speedata: speedata.io/technology, read 10 Nov 2026
  11. Speedata: speedata.io/workload-analyzer, read 10 Nov 2026
  12. Speedata: github.com/Speedata-io, read 10 Nov 2026
  13. Apache DataFusion Comet GitHub repository, read 10 Nov 2026

Last reviewed 10 Nov 2026.