Skip to content
Spark Acceleration Compare

Guide, read 10 Nov 2026

Native Spark execution engines, explained

All guidesOpen the Ledger

What changes inside the executor

In stock Spark, a stage runs as generated JVM code, with rows held as Java objects or off-heap Tungsten buffers and memory managed by the garbage collector. A native engine keeps Catalyst as the planner, then hands a whole subtree of the physical plan to a compiled library. The library processes columns in batches, so one function call handles thousands of values and the CPU can use vector instructions. Data stays in columnar buffers outside the JVM heap.

The cost is a boundary. Where a native subtree meets a JVM operator, data is converted between row and columnar formats. Each boundary adds a conversion, so a plan fragmented by unsupported operators pays that cost more often. The Comet tuning guide has a page on reducing row and columnar conversions for this reason.

The engines and their building blocks

Velox is a C++ library created by Meta and developed with IBM, Intel, Voltron Data, Microsoft, ByteDance and others. Its README says it is not meant to be used directly by end users: it takes an optimized plan and runs it. Its components include an Arrow-compatible vector layout with Flat, Dictionary, Constant and Sequence/RLE encodings, a vectorized expression evaluator, scalar, aggregate and window functions following Presto and Spark semantics, and scan, join, sort, shuffle and write operators. Velox reads ORC/DWRF, Parquet and Nimble files from S3, HDFS, GCS and ABFS.

Apache Auron (originally Blaze) is a Spark plugin with a native engine written in Rust, built on DataFusion and Arrow. You build the jar yourself with Rust nightly and a JDK.

Apache DataFusion Comet is a Spark plugin built on Apache DataFusion, a Rust query engine on Apache Arrow. Comet is Apache-2.0 licensed under the Apache Software Foundation and its Maven jars bundle native libraries for Linux on amd64 and arm64.

Apache Gluten is the middle layer that offloads Spark SQL execution to a native backend, and Velox is the backend most documented. Microsoft Fabric's native execution engine and Google's Lightning Engine are listed as users on Gluten's Powered By page, and DataPelago's documentation configures the Gluten plugin class.

Databricks Photon is a proprietary C++ vectorized engine inside Databricks Runtime, described in a SIGMOD 2022 paper. It is not available outside Databricks compute.

Flarion says its engine is built on Apache Arrow and DataFusion and is proprietary.

Microsoft Fabric native execution engine is Microsoft's packaging of Velox and Gluten for Fabric Spark runtimes 1.3 (Spark 3.5) and 2.0 (Spark 4.1).

Operator coverage and fallback

Each project documents what it does not cover. Gluten's Velox backend page lists known differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 and so have no lookaround, file scans fully support only Parquet and partly ORC, and JSON functions differ for single-quoted strings. Comet's documentation says some expressions have native implementations that differ from Spark and stay off unless allowIncompatible is set, and that Iceberg writes are off by default. Fabric lists decimal-to-float casting, timezone, round(), map() duplicate keys and collect_list ordering as differences on Runtime 1.3. Photon does not support UDFs, RDD or Dataset APIs, or stateful streaming.

The guide on operator coverage and fallback shows where each project says to look for the list.

Spark versions, memory and where each one runs

All five have the same shape: your code is unchanged, your configuration and runtime are not. Speedups published for them are listed, with who ran each test, in the Spark Accelerator Ledger.

Read 2026-11-10 from each project's documentation. Using an unofficial Spark version with Gluten can throw NoSuchMethodError at runtime, per its docs.
Engine (alphabetical)Spark versions documentedWhere it runsMemory note
Apache Auron3.0 to 4.2 per its build scriptSpark on YARN in its benchmark; no EMR, EKS or Databricks support documentedOff-heap disabled in the quick start; memory moved into executor overhead
Apache DataFusion Comet3.5.9, 4.0.4, 4.1.3, 4.2.0; JDK 17 or laterOwn Spark clusters; benchmarked on EKS; managed forks may not fully workOff-heap pool and executor overhead must be set
Apache Gluten with VeloxOfficial: 3.4.4, 3.5.5, 4.0.2, 4.1.1Own clusters; Ubuntu 20.04/22.04 or CentOS 8Off-heap size must be enabled and set
Databricks PhotonDatabricks Runtime versionDatabricks on AWS, Azure and GCPManaged by Databricks
Microsoft Fabric native execution engineRuntime 1.3 and 2.0Fabric (Azure) onlyManaged by Fabric

What the native projects have published

The Apache Gluten homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: 3.34x overall on TPC-H (up to 23.45x on a single query) and 3.02x on TPC-DS (up to 13.75x on a single query). It links no method, and the hardware is from Intel, a contributor. The Apache DataFusion Comet project publishes TPC-DS-derived results at scale factor 1000 on EKS with its scripts and every non-default setting. Apache Auron says about 2x faster and about 50% fewer cluster resources against Spark 3.5 on a TPC-DS 1 TB run it describes end to end, with results shown as a chart. Databricks publishes a 3x average and a more-than-10x maximum for Photon against its own earlier runtime.

These numbers come from different baselines, scales, hardware and Spark versions. Do not subtract one from another. The guide on reading a vendor benchmark lists the checks that make two results comparable.

Sources

  1. Velox README, read 10 Nov 2026
  2. Apache Gluten: Velox backend limitations, read 10 Nov 2026
  3. Apache Gluten: Powered By, read 10 Nov 2026
  4. Apache DataFusion Comet: compatibility guide, read 10 Nov 2026
  5. Apache DataFusion Comet: installation, read 10 Nov 2026
  6. Apache Auron, read 10 Nov 2026
  7. Photon paper, SIGMOD 2022, read 10 Nov 2026
  8. Microsoft Learn: Native execution engine for Fabric Spark, read 10 Nov 2026
  9. Flarion about page, read 10 Nov 2026

Last reviewed 10 Nov 2026.