Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Apache Auron and Databricks Photon: the published facts in the same rows

Apache Auron profileDatabricks Photon profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowApache AuronDatabricks Photon
KindOpen-source Rust engine plugin for SparkDatabricks's C++ vectorized engine
Does wellThe project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. Its build script lists Spark 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13.Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021.
LimitedYou build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. Support for Amazon EMR, EKS or Databricks: none found on 2026-11-10 in the project's pages. The benchmark's per-query results are published as a chart image only.Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.
Published speed resultsThe project says it has passed all TPC-DS and TPC-H cases and, compared with Spark 3.5, runs about 2x faster and saves about 50% of cluster resources. The only published detail is the TPC-DS 1 TB benchmark page; no per-query numbers are in text form (results are a chart).Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job.
Published cost results and pricesNone found on 10 Nov 2026.Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing.
Code changesAuron is added by putting a jar on the Spark classpath and setting spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager; no job changes. The project describes fine-grained fallback for unsupported operators, and the Spark UI has an Auron tab that records per-operator fallback reasons. You must build the jar yourself (Rust nightly and JDK required); the getting-started page describes building from source.Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.
TuningThe suggested config disables Spark off-heap memory and moves memory into executor overhead (4096 MB in the quick start). The benchmark used executor memory 8g with overhead 16384 for Auron versus 20g with 4096 for plain Spark, plus spark.auron.memoryFraction 0.8, forceShuffledHashJoin and sort-merge-join and UDAF fallback settings. ORC with large string/binary values needs spark.auron.orc.batchSize lowered.Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback.
Spark versions, platforms and table formatsThe build script lists supported Spark versions 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1 and 4.2 with Scala 2.12 and 2.13, and Celeborn 0.5/0.6 and Uniffle 0.10 remote shuffle. The 8.0.0 notes add Spark 4.0 and 4.1 build support. No EMR, EKS or Databricks support is documented. The 8.0.0-incubating release notes say it added and expanded native scan support for Apache Iceberg, Apache Hudi and Apache Paimon, including Copy-on-Write tables and Iceberg changelog scans. The homepage says it supports Hudi and Paimon. 8.0.0 also adds Flink integration. Delta Lake is not mentioned in the fetched pages.Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark.
Where it runsBuild needs Rust nightly and JDK; the README says regularly tested on JDK 8, 11 and 17 and also on JDK 21. The build script supports OS images centos7, ubuntu24, rockylinux8, debian11 and azurelinux3, and macOS builds were added in 8.0.0. The homepage says it supports storage such as HDFS and S3. No cloud-specific instance guidance was found.Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses.
How open its benchmarks areThe benchmark page publishes the method end to end: TPC-DS 1 TB data generated with a datagen tool (derived from maropu/spark-tpcds-datagen) and run with a benchmark runner (derived from databricks/spark-sql-perf), both in the auron-project/tpcds-tools repo, with the full Spark config. It ran Spark 3.5.6 against Auron 6.0.0-preview on YARN. Node types and counts are not given (the config shows executor cores 8 and a very large maxExecutors).The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found.
Change Budget: codeNo job code change stated. Auron is added with a jar on the Spark classpath and three settings (spark.auron.enable, the AuronSparkSessionExtension and the AuronShuffleManager); the project states no job changes.No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.
Change Budget: clusterBuild the jar, set three configs. You must build the jar yourself (Rust nightly and a JDK). The quick start disables Spark off-heap memory and moves memory into executor overhead (4096 MB).Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate.
Change Budget: cloudNone found on 2026-11-10. No cloud requirement is stated. The homepage says it supports storage such as HDFS and S3, and no cloud-specific instance guidance was found. Documented support for EMR, EKS or Databricks: none found on 2026-11-10.Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud.
Change Budget: vendorApache-2.0, no contract stated. Auron is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described.Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page.
Ledger entriesAuron TPC-DS 1 TB against Apache Spark 3.5.6 (run by its maker)Photon customer speedups, SIGMOD 2022 paper (run by its maker); Databricks audited 100 TB TPC-DS result on Photon (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. Apache Auron (incubating), formerly Blaze: auron.apache.org, read 10 Nov 2026
  2. Apache Auron (incubating), formerly Blaze: auron.apache.org/documents/benchmarks.html, read 10 Nov 2026
  3. Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/auron-build.sh, read 10 Nov 2026
  4. Apache Auron (incubating), formerly Blaze: auron.apache.org/archives/v8.0.0-incubating.html, read 10 Nov 2026
  5. Apache Auron (incubating), formerly Blaze: raw.githubusercontent.com/apache/auron/master/README.md, read 10 Nov 2026
  6. Databricks Photon: docs.databricks.com/en/compute/photon.html, read 10 Nov 2026
  7. Databricks Photon: databricks.com/product/pricing, read 10 Nov 2026
  8. Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf, read 10 Nov 2026
  9. Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing, read 10 Nov 2026

Last reviewed 10 Nov 2026.