Side by side, read 10 Nov 2026
Apache Gluten with Velox and Databricks Photon: the published facts in the same rows
Apache Gluten with Velox profileDatabricks Photon profile
The same rows for both
| Row | Apache Gluten with Velox | Databricks Photon |
|---|---|---|
| Kind | Open-source Spark plugin offloading to Velox | Databricks's C++ vectorized engine |
| Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021. |
| Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. |
| Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | Docs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job. |
| Published cost results and prices | None found on 10 Nov 2026. | Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing. |
| Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Docs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. |
| Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | Photon is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback. |
| Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | Photon accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark. |
| Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Photon is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses. |
| How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found. |
| Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported. |
| Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Set runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Databricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud. |
| Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Databricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page. |
| Ledger entries | Gluten with Velox, TPC-DS 3 TB (project homepage) (run by its maker); Gluten with Velox, TPC-H 3 TB (project homepage) (run by its maker) | Photon customer speedups, SIGMOD 2022 paper (run by its maker); Databricks audited 100 TB TPC-DS result on Photon (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Apache Gluten with the Velox backend: gluten.apache.org, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: gluten.apache.org/downloads, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md, read 10 Nov 2026
- Databricks Photon: docs.databricks.com/en/compute/photon.html, read 10 Nov 2026
- Databricks Photon: databricks.com/product/pricing, read 10 Nov 2026
- Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf, read 10 Nov 2026
- Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing, read 10 Nov 2026
- Apache Gluten Powered By page, read 10 Nov 2026
Last reviewed 10 Nov 2026.