# DataFusion

Glossary, read 10 Nov 2026

**The short answer.** Apache DataFusion is a query engine written in Rust that uses the Apache Arrow in-memory format. It offers SQL and DataFrame APIs with built-in support for CSV, Parquet, JSON and Avro. It began inside the Arrow project and now lists subprojects including Ballista, Python bindings and Comet.

DataFusion is a toolkit for building engines, not a Spark replacement. Comet takes DataFusion's native operators and plugs them into Spark's execution path through the JVM.

Flarion, a commercial plugin, also says it is built on Arrow and DataFusion. The two products differ in license and support terms, so read each project's own documents.

## A sourced example

The Comet project's TPC-DS run at scale factor 1000 on an EKS cluster reported 994 s on Spark 4.2.0 and 488 s on Comet 1.2.0; the project ran it, so read it as a project number.

## Sources

- Apache DataFusion user guide: Introduction: https://datafusion.apache.org/user-guide/introduction.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/glossary/datafusion/
