Side by side, read 10 Nov 2026
Google Cloud Lightning Engine and Speedata: the published facts in the same rows
Google Cloud Lightning Engine profileSpeedata profile
The same rows for both
| Row | Google Cloud Lightning Engine | Speedata |
|---|---|---|
| Kind | Premium Spark engine on Google Cloud | PCIe analytics chip plus Dash Spark plugin |
| Does well | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. Speedata offers a free Workload Analyzer that takes Spark logs and predicts per-query acceleration before purchase. |
| Limited | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. No price list is published, and EMR, EKS, Databricks and table format support are not stated on the pages read. |
| Published speed results | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. | The home page claims the APU accelerates Apache Spark, batch ETL and AI data preparation up to 100x, with up to 90% lower TCO and zero code changes. Customer-result tiles show 62.7x acceleration on production Spark workloads (37 servers consolidated to 3), 90 to 8 hours of processing time on the same hardware budget, and 52x more performance per dollar than a GPU on analytics queries on an 'industry-standard benchmark'. No customer names, data sizes or method are given for these tiles. Speedata's blog says Microsoft's SIGMOD 2026 industry paper award 'CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric' benchmarked the Speedata C-200 APU as one of three coprocessors. Per Speedata's summary of Table 2, standalone APU speedups on TPC-H SF100 (Parquet, warm in CPU memory) versus single-core SQL Server on an Intel Xeon Platinum 8473C were: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x. This is a selection of five queries, run against SQL Server on one core, not against Spark. The paper page on microsoft.com could not be fetched, so these numbers come from Speedata's blog only. The product page shows an example Workload Analyzer result of a query going from 4 minutes 3 seconds to 13.0 seconds (20X). It is presented as an illustration of the tool, with no workload details. |
| Published cost results and prices | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | No price list is published. The pages offer buying a server or going through OEM or cloud partners. The only cost claims are up to 90% lower TCO, 37 servers replaced by 3, and 52x performance per dollar versus a GPU. Hardware must be bought or rented, so a comparison with pay-per-use cloud plugins needs a quote. |
| Code changes | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | Dash software plugs into the Spark Catalyst optimizer, identifies compute-intensive work and offloads it to the APU. SQL elements that cannot be accelerated, such as UDFs, are delegated to the CPU while the rest of the query runs on the APU. The page shows enabling it with a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. |
| Tuning | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. | The pages say zero code changes and a transparent plug-in. No per-job tuning guide or list of required Spark settings beyond the plugin class and spark.speedata.apus is published on the pages read. A free Workload Analyzer (upload Spark logs, run a CLI locally, or compare against TPC-DS) predicts per-query acceleration before purchase. |
| Spark versions, platforms and table formats | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. | The technology page says Dash accelerates Apache Spark 3.x workloads on Kubernetes, YARN and standalone cluster managers. The example command uses speedata-dash-0.8.1-spark_2.12.jar, and the company GitHub org has a container image repo built on Spark 3.5.2. The scope described is Spark SQL, DataFrame and batch ETL. EMR, EKS, Databricks, Iceberg, Delta and Hudi support are not stated on the pages read. Spark 4 is not mentioned. |
| Where it runs | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). | The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Options on the home page: a pre-configured 2U server with two C200 cards from Speedata; servers from OEM partners Dell and HPE; and 'Speedata Acceleration for AWS', where the customer keeps data in S3, syncs deltas to a Speedata-hosted environment, runs the work on the APU there and pushes results back. That means Speedata does not run inside the customer's own EC2 fleet. A European sovereign-cloud partner, Nebul, was announced on Feb 19, 2026. |
| How open its benchmarks are | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. | The vendor's customer-result tiles have no published method. The GitHub org Speedata-io has five repositories, including a TPC-H data and query generator and a Spark 3.5.2 container image, but no repo with the benchmark results or APU software. A technical white paper is behind a form and was not read. |
| Change Budget: code | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. | No code change stated. Speedata says zero code changes. The Dash plugin identifies compute-intensive work and offloads it, delegating UDFs to the CPU. |
| Change Budget: cluster | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. | PCIe C200 card in each server. The APU is a PCIe Gen 5 card (C200 with the Callisto ASIC), so it needs physical servers. Enabling it uses a jar and two settings: spark.plugins=com.speedata.spark.DashPlugin and spark.speedata.apus=1. |
| Change Budget: cloud | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). | On-premises, OEM or hosted AWS. Options are a 2U server with two C200 cards, OEM servers from Dell and HPE, or Speedata Acceleration for AWS, which runs the work in a Speedata-hosted environment and syncs data from S3. |
| Change Budget: vendor | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. | Speedata purchase; no price list. No price list is published. The pages offer buying a server or going through OEM or cloud partners. |
| Ledger entries | Lightning Engine against open source Spark, TPC-H-like 10 TB (run by its maker); Lightning Engine claim on Google's pricing page (run by its maker); Native Query Execution on Axion ARM, Google's documentation (run by its maker) | Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper (run by Microsoft (paper); numbers read from Speedata's blog); Speedata production Spark workload tile (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, read 10 Nov 2026
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing, read 10 Nov 2026
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, read 10 Nov 2026
- Speedata: speedata.io, read 10 Nov 2026
- Speedata: speedata.io/post/259-faster-what-microsoft-s-award-winning-sigmod-paper-, read 10 Nov 2026
- Speedata: speedata.io/product, read 10 Nov 2026
- Speedata: speedata.io/technology, read 10 Nov 2026
- Speedata: speedata.io/workload-analyzer, read 10 Nov 2026
- Speedata: github.com/Speedata-io, read 10 Nov 2026
Last reviewed 10 Nov 2026.