# Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine: the published facts in the same rows

Side by side, read 10 Nov 2026

**The short answer.** Amazon EMR runtime for Apache Spark and Google Cloud Lightning Engine are set out here in the same rows: published speed and cost results, code and cluster changes, tuning, stack fit and where each runs. The Ledger holds 1 entry for Amazon EMR runtime for Apache Spark and 3 entries for Google Cloud Lightning Engine.

## The same rows for both

| Row | Amazon EMR runtime for Apache Spark | Google Cloud Lightning Engine |
| --- | --- | --- |
| Kind | AWS-tuned Spark runtime on EMR | Premium Spark engine on Google Cloud |
| Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. |
| Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. |
| Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. |
| Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. |
| Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. |
| Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). |
| How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. |
| Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. |
| Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). |
| Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Ledger entries | [EMR runtime claim on the AWS features page](/ledger/amazon-emr-runtime-features-page-claim/) (run by its maker) | [Lightning Engine against open source Spark, TPC-H-like 10 TB](/ledger/google-lightning-engine-tpch-like-10tb/) (run by its maker); [Lightning Engine claim on Google's pricing page](/ledger/google-lightning-engine-pricing-page-claim/) (run by its maker); [Native Query Execution on Axion ARM, Google's documentation](/ledger/google-native-query-execution-axion/) (run by its maker) |

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.

## How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The [Ledger method](/ledger/method/) explains how entries are recorded.

## Sources

- Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026)
- Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark: https://github.com/aws-samples/emr-spark-benchmark (read 10 Nov 2026)
- Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026)
- Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026)
- Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi: https://cloud.google.com/blog/products/data-analytics/introducing-lightning-engine-for-apache-spark (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/compare/dataproc-lightning-engine-vs-emr-spark-runtime/
