Skip to content
Spark Acceleration Compare

Option profile, read 10 Nov 2026

Google Cloud Lightning Engine, Premium Spark engine on Google Cloud: the published facts

Google Cloud Lightning Engine in the LedgerCompare with another option

Kind
Premium Spark engine on Google Cloud
Website
docs.cloud.google.com/managed-spark/docs/guides/lightning-engine
Name checked
Lightning Engine still exists, but the host service is now named 'Managed Service for Apache Spark' (Dataproc and Serverless for Apache Spark docs now redirect to docs.cloud.google.com/managed-spark). Serverless premium tier is 'Powered by Lightning Engine'. Native Query Execution (NQE, Gluten and Velox) is now an optional component of Lightning Engine, enabled on clusters.
Read
10 Nov 2026

What it does well

  • Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning.
  • Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second.

Where it is limited

  • Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings.
  • Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail.

Published speed results

FactSource typeSource, read
Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results.vendorcloud.google.com/blog/products/data-analytics/introducing-lightning-engi, 10 Nov 2026
The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page.vendor docsdocs.cloud.google.com/managed-spark/pricing, 10 Nov 2026
The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page.vendor docsdocs.cloud.google.com/managed-spark/docs/guides/lightning-engine, 10 Nov 2026

Published cost results and prices

FactSource typeSource, read
Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.vendor docsdocs.cloud.google.com/managed-spark/pricing, 10 Nov 2026

Code changes

FactSource typeSource, read
Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.vendor docsdocs.cloud.google.com/managed-spark/docs/guides/lightning-engine, 10 Nov 2026

Tuning

FactSource typeSource, read
NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs.vendor docsdocs.cloud.google.com/managed-spark/docs/guides/lightning-engine, 10 Nov 2026

Spark versions, platforms and table formats

FactSource typeSource, read
NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines.vendor docsdocs.cloud.google.com/managed-spark/docs/guides/lightning-engine, 10 Nov 2026

Where it runs

FactSource typeSource, read
Google Cloud only. Rates on the pricing page are regional (us-central1 example given).vendor docsdocs.cloud.google.com/managed-spark/pricing, 10 Nov 2026

How open its benchmarks are

FactSource typeSource, read
The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings.vendorcloud.google.com/blog/products/data-analytics/introducing-lightning-engi, 10 Nov 2026

Profile

FactSource typeSource, read
Lightning Engine is a high-performance execution environment for Apache Spark on Managed Service for Apache Spark that accelerates compute-intensive workloads without requiring changes to Spark code. It targets DataFrame, Dataset and Spark SQL workloads; I/O-bound or RDD-heavy jobs may suit the standard engine.vendor docsdocs.cloud.google.com/managed-spark/docs/guides/lightning-engine, 10 Nov 2026
Google announced Lightning Engine in preview on May 30, 2025.vendorcloud.google.com/blog/products/data-analytics/introducing-lightning-engi, 10 Nov 2026

Ledger entries for Google Cloud Lightning Engine

PublishedOptionTestResult as publishedRan byMethod published
2025-05-30Google Cloud Lightning EngineLightning Engine against open source Spark, TPC-H-like 10 TB3.6x acceleration over open source SparkGoogle Cloud (its maker)No
Not statedGoogle Cloud Lightning EngineLightning Engine claim on Google's pricing pageGoogle Cloud's wording: up to 4.9x faster than open source Spark; up to 2.5x the price-performance of a high-speed Spark alternative that the page does not nameGoogle Cloud (its maker)No
Not statedGoogle Cloud Lightning EngineNative Query Execution on Axion ARM, Google's documentationGoogle Cloud's wording: 2.5x to 4x speedup over open source SparkGoogle Cloud (its maker)No

Change Budget row

ColumnWhat changesDetailSource, read
CodeNo Spark code change statedGoogle's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine.Google Lightning Engine documentation, 10 Nov 2026
ClusterImage 2.3.3+ or 3.0.2+, engine flagClusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail.Google Lightning Engine documentation, 10 Nov 2026
CloudGoogle Cloud onlyGoogle lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example).Google Managed Service for Apache Spark pricing, 10 Nov 2026
VendorPremium tier or cluster add-onServerless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks.Google Managed Service for Apache Spark pricing, 10 Nov 2026

Searched for and none found on 10 Nov 2026

  • Independent third-party benchmark of Lightning Engine (searched: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, docs.cloud.google.com/managed-spark/pricing)

Google Cloud Lightning Engine side by side with each other option

Sources

  1. Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, read 10 Nov 2026
  2. Google Cloud Lightning Engine: cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, read 10 Nov 2026
  3. Google Cloud Lightning Engine: docs.cloud.google.com/managed-spark/pricing, read 10 Nov 2026

Last reviewed 10 Nov 2026.