Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

How to read a vendor Spark benchmark

All guidesOpen the Ledger

Check 1: who ran it

A result run by the vendor, a cloud provider, the vendor and a cloud provider together, or a third party are four different kinds of evidence. The Spark Accelerator Ledger on this site records who ran each entry and whether that party sells the product. Nothing is averaged across them.

Examples from the published record, read 2026-11-10: AWS and NVIDIA ran the 3 TB GPU test on Amazon EMR on EKS together, and NVIDIA published a separate comparison against open-source Spark on EC2. The Apache Auron and Apache DataFusion Comet projects run their own tests on their own published tooling. Databricks ran its audited 100 TB TPC-DS record on Databricks SQL. Every number DualBird has published so far is DualBird's own. Flarion ran a six-engine comparison on its own page and discloses that its own engine is slower than Databricks Photon in both scales.

Check 2: TPC-DS and TPC-H derived versus real workloads

The TPC benchmarks are fixed query sets. An audited TPC result follows the council's rules and comes with a full report of the run. A "TPC-DS derived" or "TPC-H-like" run does not, and the labels say so. NVIDIA's NDS repository states that NDS is derived from TPC-DS and is not comparable to published TPC-DS results. Google's Lightning Engine launch post says the same of its TPC-H-derived queries.

Real workloads are what you run, but a customer tile with no workload attached cannot be checked. DataPelago's site carries customer testimonials with 50% cost reductions. Speedata's home page shows 62.7x on production Spark workloads with 37 servers consolidated to 3. Neither names a baseline, data size or method. Treat them as claims, not measurements.

Checks 3 to 5: scale factor, hardware and baseline

Scale factor. SF100 is about 100 GB of raw data, SF1000 about 1 TB. Flarion's comparison runs both on one i4i.8xlarge: Spark 3.5.5 took 1,135 seconds at SF100 and 135.6 minutes at SF1000, so the scale changes the story. Query counts differ too: the AWS and NVIDIA test ran 103 queries, the AWS sample harness runs 104.

Hardware. Ask for instance type, count, storage and region. The Auron benchmark page publishes the Spark settings and the tooling but not the node type or count. Comet's results page names the r6i family on EKS but not the instance size. The Gluten homepage reports 3.34x on TPC-H and 3.02x on TPC-DS at 3 TB, on Intel Xeon Platinum 8592+ with Spark 3.3.1, without cluster size or scripts.

Baseline version and settings. This is the check that moves results most. On the same tables and settings, Comet's results page reports Spark 4.2.0 at 994 seconds, Comet 1.2.0 at 488 seconds (2.0x), and Spark 4.1.3 at 1,336 seconds with Comet 1.2.0 at 532 seconds (2.51x). The project explains the gap: Spark 4.1.3 is slower mainly because of the default S3A input stream. Spark 4.2.0 with the classic stream took 1,294 seconds. A 2.0x and a 2.51x claim come from the same engine and the same page. Also check what the baseline was tuned like: Auron's run gave plain Spark 20g executor memory with 4096 overhead and its own engine 8g with 16384 overhead.

Checks 6 and 7: cost basis and published method

A speedup is not a cost result. AWS's GPU post lists $26.35 per hour for the 8-node g7.4xlarge cluster against $10.43 to $10.99 for the CPU clusters, and arrives at $2.06 per run against $2.93 to $3.18 using On-Demand us-east-1 prices as of May 2026. It leaves out the EKS control-plane fee, EBS, S3 and the driver pod. Photon instances consume DBUs at a different rate than the same instance without Photon, so Databricks's cost depends on the multiplier on its pricing page. NVIDIA's own overview compares a different CPU run and states that the stacks differ. Check whether a quoted saving includes license fees: DualBird publishes no plugin price, so a reader cannot tell whether its 55 to 85% lower EC2 cost figure includes one (DualBird's figure, from its Iceberg compaction post).

A published method means scripts, configuration and data generation that you could rerun. AWS publishes an emr-spark-benchmark repository with commands but no results. Auron publishes the datagen tool and runner in a public repository. Comet publishes benchmark scripts and every non-default setting. DualBird's compaction methodology is in a download behind a form, and its enrichment tile (110 GB, 6x to 18x) gives no method at all. Flarion lists its caps and partition counts and says q72 is counted at vanilla Spark's time for Gluten and Comet.

A short worksheet

Where DualBird's benchmarks are described on this site, the same questions are asked. DualBird is available on AWS F2 instances only, not on Azure, Google Cloud or Databricks-managed clusters, and every published DualBird benchmark is DualBird's own.

Entries in the Spark Accelerator Ledger are recorded in this order.
QuestionWhere the answer should be
Who ran it and do they sell the product?Byline or repository owner
TPC-DS, TPC-H, derived, or your kind of job?Workload line and the footnotes
Scale factor and query count?Results table or config
Instance type, count, region?Hardware paragraph
Baseline Spark version and its settings?Config files in the repository
Cost per run or per job, with which prices and date?Cost table
Can you rerun it?Scripts linked from the page

Sources

  1. Apache DataFusion Comet: TPC-DS benchmark results, read 10 Nov 2026
  2. AWS Big Data Blog: GPU-accelerated Apache Spark on Amazon EMR with G7 instances, read 10 Nov 2026
  3. NVIDIA cuDF plugin for Apache Spark: overview, read 10 Nov 2026
  4. NVIDIA NDS benchmark README, read 10 Nov 2026
  5. Apache Auron: benchmarks, read 10 Nov 2026
  6. Apache Gluten homepage, read 10 Nov 2026
  7. Flarion benchmarks, read 10 Nov 2026
  8. AWS samples: emr-spark-benchmark, read 10 Nov 2026
  9. Google Cloud blog: Lightning Engine for Apache Spark, read 10 Nov 2026
  10. DualBird: Iceberg compaction benchmark, read 10 Nov 2026
  11. Speedata home page, read 10 Nov 2026

Last reviewed 10 Nov 2026.