Glossary, read 10 Nov 2026
DataFusion
DataFusion is a toolkit for building engines, not a Spark replacement. Comet takes DataFusion's native operators and plugs them into Spark's execution path through the JVM.
Flarion, a commercial plugin, also says it is built on Arrow and DataFusion. The two products differ in license and support terms, so read each project's own documents.
A sourced example
The Comet project's TPC-DS run at scale factor 1000 on an EKS cluster reported 994 s on Spark 4.2.0 and 488 s on Comet 1.2.0; the project ran it, so read it as a project number.
Sources
- Apache DataFusion user guide: Introduction, read 10 Nov 2026
Last reviewed 10 Nov 2026.