Lightning Engine still exists, but the host service is now named 'Managed Service for Apache Spark' (Dataproc and Serverless for Apache Spark docs now redirect to docs.cloud.google.com/managed-spark). Serverless premium tier is 'Powered by Lightning Engine'. Native Query Execution (NQE, Gluten and Velox) is now an optional component of Lightning Engine, enabled on clusters.
Read
10 Nov 2026
What it does well
Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning.
Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second.
Where it is limited
Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings.
Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail.
Published speed results
Fact
Source type
Source, read
Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results.
The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page.
The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page.
Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.
Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.
NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs.
NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines.
Lightning Engine is a high-performance execution environment for Apache Spark on Managed Service for Apache Spark that accelerates compute-intensive workloads without requiring changes to Spark code. It targets DataFrame, Dataset and Spark SQL workloads; I/O-bound or RDD-heavy jobs may suit the standard engine.
Google Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not name
Google Cloud's wording: 2.5x to 4x speedup over open source Spark
Google Cloud (its maker)
No
Change Budget row
Column
What changes
Detail
Source, read
Code
No Spark code change stated
Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.
Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail.
Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.