Skip to content
Spark Acceleration Compare

Ledger entry, published 2026-11-03

Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000

All Ledger entriesApache DataFusion Comet profile

Option tested
Apache DataFusion Comet
Ran by
Apache DataFusion Comet project (the maker of the option tested)
Published
2026-11-03
Workload
TPC-DS derived queries, 103 queries, Parquet (zstd) in S3
Data size
1 TB (scale factor 1000)
Hardware
Same EKS cluster on r6i instances as the Spark 4.2.0 run
Result as published
Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x)
Cost result as published
None published
Method published
Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository

What the test shows and does not show

Spark 4.2.0 with the classic S3A stream took 1,294 s on the same cluster, according to the project.

The two baseline rows together show how far a speedup ratio moves with the Spark version.

Other Ledger entries for Apache DataFusion Comet

Sources

  1. Comet TPC-DS benchmark results, read 10 Nov 2026

Last reviewed 10 Nov 2026.