Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Databricks Photon and Google Cloud Lightning Engine: the published facts in the same rows

Databricks Photon profileGoogle Cloud Lightning Engine profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowDatabricks PhotonGoogle Cloud Lightning Engine
KindDatabricks's C++ vectorized enginePremium Spark engine on Google Cloud
Does wellPer Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. Databricks announced a TPC-audited 100 TB TPC-DS result of 32,941,245 QphDS in November 2021.Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second.
LimitedPhoton is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. Per the same documentation, Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail.
Published speed resultsDocs claim up to 5x better price/performance for data and analytics workloads compared to other cloud data warehouses, as measured by industry-standard TPC-DS benchmarks. This compares against warehouses, not against open source Spark. The SIGMOD 2022 Photon paper reports customers observed average speedups of 3x over the previous Databricks Runtime and maximum speedups of over 10x. The baseline is Databricks' own prior Spark-based runtime, not open source Spark. Databricks announced an audited 100TB TPC-DS record in November 2021: 32,941,245 QphDS versus 14,861,137 for the previous record holder (Alibaba). The post says Databricks SQL 'is not based on Apache Spark, but rather Photon'. The run was on Databricks SQL, not a Spark batch job.Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page.
Published cost results and pricesPhoton instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page, not only on speedup. Databricks defines a DBU as a normalized unit of processing power used for pricing.Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.
Code changesDocs say Photon works with existing Spark API code with no changes. Unsupported operations fall back transparently to the Spark runtime for the remainder of that operation. Photon does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.
TuningPhoton is on by default for classic all-purpose, jobs compute and classic pipelines (checkbox Use Photon Acceleration), and enabled on serverless and SQL warehouses. If you use the Clusters or Jobs API you must set runtime_engine to PHOTON. Queries under about two seconds see no meaningful gain. Users check the Spark UI to see fallback.NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs.
Spark versions, platforms and table formatsPhoton accelerates Delta Lake, Apache Iceberg and Parquet writes, supports SQL and DataFrame APIs in Python, R, Scala and Java, and stateless streaming from Delta, Parquet, CSV, JSON, Kafka and Kinesis. It runs on Databricks Runtime, so the Spark version is the Databricks Runtime version, not a user choice of open source Spark.NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines.
Where it runsPhoton is documented only as part of Databricks compute on AWS, Azure and GCP (cloud tabs on the page). It is not offered as a standalone engine for open source Spark, EMR or other platforms in the docs read. Photon supports a set of driver and worker instance types including Graviton. Graviton with Photon does not support Databricks Container Services for dedicated compute or SQL warehouses.Google Cloud only. Rates on the pricing page are regional (us-central1 example given).
How open its benchmarks areThe TPC-DS record is a TPC-audited result, so a full report exists under TPC rules, but it needs a Databricks SQL warehouse and is not a repeatable open script. No public repo for Photon speedups on user Spark jobs was found.The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings.
Change Budget: codeNo change, with listed exceptions. Databricks says Photon works with existing Spark API code with no changes. It does not support UDFs, RDD APIs or Dataset APIs, and stateful streaming is not supported.No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.
Change Budget: clusterSet runtime_engine to PHOTON in API. Photon is on by default for classic all-purpose, jobs compute and classic pipelines. With the Clusters or Jobs API you must set runtime_engine to PHOTON. Photon instance types use a different DBU rate.Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail.
Change Budget: cloudDatabricks on AWS, Azure or GCP. Photon is documented as part of Databricks compute on AWS, Azure and Google Cloud.Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example).
Change Budget: vendorDatabricks platform required. Photon runs only inside Databricks. Photon instance types consume DBUs at a different rate than the same instance without Photon, so cost depends on the DBU multiplier on the pricing page.Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.
Ledger entriesPhoton customer speedups, SIGMOD 2022 paper (run by its maker); Databricks audited 100 TB TPC-DS result on Photon (run by its maker)Lightning Engine against open source Spark, TPC-H-like 10 TB (run by its maker); Lightning Engine claim on Google's pricing page (run by its maker); Native Query Execution on Axion ARM, Google's documentation (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. Databricks Photon: docs.databricks.com/en/compute/photon.html, read 10 Nov 2026
  2. Databricks Photon: databricks.com/product/pricing, read 10 Nov 2026
  3. Databricks Photon: people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf, read 10 Nov 2026
  4. Databricks Photon: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing, read 10 Nov 2026
  5. Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, read 10 Nov 2026
  6. Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing, read 10 Nov 2026
  7. Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, read 10 Nov 2026

Last reviewed 10 Nov 2026.