Ledger entry, published 2026-11-03
Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000
All Ledger entriesApache DataFusion Comet profile
- Option tested
- Apache DataFusion Comet
- Ran by
- Apache DataFusion Comet project (the maker of the option tested)
- Published
- 2026-11-03
- Workload
- TPC-DS derived queries, 103 queries, Parquet (zstd) in S3
- Data size
- 1 TB (scale factor 1000)
- Hardware
- Same EKS cluster on r6i instances as the Spark 4.2.0 run
- Result as published
- Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x)
- Cost result as published
- None published
- Method published
- Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
What the test shows and does not show
Spark 4.2.0 with the classic S3A stream took 1,294 s on the same cluster, according to the project.
The two baseline rows together show how far a speedup ratio moves with the Spark version.
Other Ledger entries for Apache DataFusion Comet
- Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF10002026-11-03, run by Apache DataFusion Comet project
Sources
- Comet TPC-DS benchmark results, read 10 Nov 2026
Last reviewed 10 Nov 2026.