# Google Cloud Lightning Engine, Premium Spark engine on Google Cloud: the published facts

Option profile, read 10 Nov 2026

**The short answer.** Lightning Engine is Google Cloud's accelerated execution environment in Managed Service for Apache Spark, announced in preview on 2025-05-30. Google's launch blog reports 3.6x on TPC-H-like queries at 10 TB versus open source Spark on similar infrastructure. In us-central1, serverless premium costs $0.089 per DCU-hour against $0.06 for standard.

- **Kind:** Premium Spark engine on Google Cloud
- **Website:** [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine)
- **Name checked:** Lightning Engine still exists, but the host service is now named 'Managed Service for Apache Spark' (Dataproc and Serverless for Apache Spark docs now redirect to docs.cloud.google.com/managed-spark). Serverless premium tier is 'Powered by Lightning Engine'. Native Query Execution (NQE, Gluten and Velox) is now an optional component of Lightning Engine, enabled on clusters.
- **Read:** 10 Nov 2026

## What it does well

- Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning.
- Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second.

## Where it is limited

- Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings.
- Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail.

## Published speed results

| Fact | Source type | Source, read |
| --- | --- | --- |
| Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 |
| The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |
| The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |

## Published cost results and prices

| Fact | Source type | Source, read |
| --- | --- | --- |
| Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |

## Code changes

| Fact | Source type | Source, read |
| --- | --- | --- |
| Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |

## Tuning

| Fact | Source type | Source, read |
| --- | --- | --- |
| NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |

## Spark versions, platforms and table formats

| Fact | Source type | Source, read |
| --- | --- | --- |
| NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |

## Where it runs

| Fact | Source type | Source, read |
| --- | --- | --- |
| Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | vendor docs | [docs.cloud.google.com/managed-spark/pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |

## How open its benchmarks are

| Fact | Source type | Source, read |
| --- | --- | --- |
| The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 |

## Profile

| Fact | Source type | Source, read |
| --- | --- | --- |
| Lightning Engine is a high-performance execution environment for Apache Spark on Managed Service for Apache Spark that accelerates compute-intensive workloads without requiring changes to Spark code. It targets DataFrame, Dataset and Spark SQL workloads; I/O-bound or RDD-heavy jobs may suit the standard engine. | vendor docs | [docs.cloud.google.com/managed-spark/docs/guides/lightning-engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| Google announced Lightning Engine in preview on May 30, 2025. | vendor | [cloud.google.com/blog/products/data-analytics/introducing-lightning-engi](https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark), 10 Nov 2026 |

## Ledger entries for Google Cloud Lightning Engine

| Published | Option | Test | Result as published | Ran by | Method published |
| --- | --- | --- | --- | --- | --- |
| 2025-05-30 | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No |
| Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) | Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name | Google Cloud (its maker) | No |
| Not stated | [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) | Google Cloud's wording: 2.5x to 4x speedup over open source Spark | Google Cloud (its maker) | No |

## Change Budget row

| Column | What changes | Detail | Source, read |
| --- | --- | --- | --- |
| Code | No Spark code change stated | Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| Cluster | Image 2.3.3+ or 3.0.2+, engine flag | Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| Cloud | Google Cloud only | Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |
| Vendor | Premium tier or cluster add-on | Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |

## Searched for and none found on 10 Nov 2026

- Independent third-party benchmark of Lightning Engine (searched: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, docs.cloud.google.com/managed-spark/pricing)

## Google Cloud Lightning Engine side by side with each other option

- [Google Cloud Lightning Engine and Amazon EMR runtime for Apache Spark](/compare/dataproc-lightning-engine-vs-emr-spark-runtime/)
- [Google Cloud Lightning Engine and Apache Auron](/compare/apache-auron-vs-dataproc-lightning-engine/)
- [Google Cloud Lightning Engine and Apache DataFusion Comet](/compare/datafusion-comet-vs-dataproc-lightning-engine/)
- [Google Cloud Lightning Engine and Apache Gluten with Velox](/compare/apache-gluten-velox-vs-dataproc-lightning-engine/)
- [Google Cloud Lightning Engine and Databricks Photon](/compare/databricks-photon-vs-dataproc-lightning-engine/)
- [Google Cloud Lightning Engine and DataPelago](/compare/datapelago-vs-dataproc-lightning-engine/)
- [Google Cloud Lightning Engine and DualBird](/compare/dataproc-lightning-engine-vs-dualbird/)
- [Google Cloud Lightning Engine and Flarion](/compare/dataproc-lightning-engine-vs-flarion/)
- [Google Cloud Lightning Engine and Microsoft Fabric native execution engine](/compare/dataproc-lightning-engine-vs-fabric-native-execution-engine/)
- [Google Cloud Lightning Engine and NVIDIA cuDF plugin for Apache Spark](/compare/dataproc-lightning-engine-vs-nvidia-rapids-spark/)
- [Google Cloud Lightning Engine and Speedata](/compare/dataproc-lightning-engine-vs-speedata/)

## Sources

- Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)
- Google Cloud Lightning Engine: cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026)
- Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/options/dataproc-lightning-engine/
