Side by side, read 10 Nov 2026
Apache Gluten with Velox and Google Cloud Lightning Engine: the published facts in the same rows
Apache Gluten with Velox profileGoogle Cloud Lightning Engine profile
The same rows for both
| Row | Apache Gluten with Velox | Google Cloud Lightning Engine |
|---|---|---|
| Kind | Open-source Spark plugin offloading to Velox | Premium Spark engine on Google Cloud |
| Does well | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. The repository includes the gluten-it tool for data generation, query runs and result comparison on Spark 3.4, 3.5, 4.0 and 4.1 profiles. | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. Google publishes rates: $0.06 per DCU-hour standard and $0.089 premium in us-central1, with serverless billed per second. |
| Limited | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. Per the Velox backend docs, Iceberg support is read-only, Hudi support covers copy-on-write tables only, and ANSI mode always falls back to vanilla Spark. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. Per Google's documentation, Native Query Execution is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs, and NQE jobs on GPU accelerators fail. |
| Published speed results | The project homepage reports TPC-H and TPC-DS at 3 TB on Intel Xeon Platinum 8592+ with Spark 3.3.1: overall 3.34x on TPC-H (up to 23.45x on single queries) and 3.02x on TPC-DS (up to 13.75x on single queries). The page does not link a method or scripts for these numbers; the hardware is from Intel, a contributor. | Google's launch blog says that at a 10 TB dataset size Lightning Engine accelerates Spark by 3.6x on TPC-H-like workloads versus open source Spark on similar infrastructure. The queries are derived from TPC-H and not comparable to official TPC-H results. The pricing page says Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name, with no workload, method or competitor named on the page. The docs claim Native Query Execution on Google Axion ARM (C4A) delivers 2.5x to 4x speedup over open source Spark on analytical workloads. No method is given on the page. |
| Published cost results and prices | None found on 10 Nov 2026. | Serverless is billed per second in Data Compute Units (DCU, 1-minute minimum). Example us-central1 rates: standard $0.06 per DCU-hour, premium (Lightning Engine) $0.089 per DCU-hour. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Code changes | Gluten is a Spark plugin that needs no DataFrame or SQL changes, only configuration: spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar. Unsupported operators fall back to vanilla Spark through columnar-to-row and row-to-columnar conversions. The Velox backend limitations page lists known fallbacks and differences: ANSI mode always falls back, case-sensitive mode may give wrong results, regexp functions use RE2 (no lookaround), file scans fully support only Parquet and partly ORC, partitioned scans need partition info in the path, and JSON functions differ for single-quoted strings. | Docs say no Spark code changes. Requires image 2.3.3+ or 3.0.2+ (debian13/ubuntu24 for Lightning Engine) and enabling with --engine=lightning at cluster creation. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Tuning | Off-heap memory must be enabled and sized (the quick start uses spark.memory.offHeap.size=20g and says to adjust it for the environment), and the columnar shuffle manager is needed for native shuffle. Docs also describe a troubleshooting page, Celeborn and Uniffle remote shuffle settings, and a setting (spark.gluten.sql.fallbackRegexpExpressions) for users who cannot tolerate regexp differences. Binary jars must load ahead of Spark's jars because Gluten overrides some Spark classes. | NQE is not recommended for RDD-heavy jobs, UDFs, most Spark ML libraries or I/O-bound jobs. NQE jobs on GPU accelerators fail. Unsupported inputs include Byte in ORC and Parquet and Struct/Array/Map in Parquet. A qualification tool exists to check jobs. |
| Spark versions, platforms and table formats | The docs state officially supported Spark versions are 3.4.4, 3.5.5, 4.0.2 and 4.1.1 (Spark 4.x needs JDK 17+ and Scala 2.13). The downloads page still lists Spark 3.3 binaries for 1.6.0 and 1.7.0, and the homepage benchmark used Spark 3.3.1. Using an unofficial Spark version can throw NoSuchMethodError at runtime. Docs do not mention EMR, EKS or Databricks support. Velox backend docs: Iceberg reads of copy-on-write and merge-on-read tables are supported but the Iceberg page says only reading is supported and only Spark 3.4 is well tested; Hudi supports reading COW tables only; Paimon supports non-primary-key tables only; Delta has a build profile and a doc page. Celeborn 0.3 to 0.5 and Uniffle 0.9.2 remote shuffle are supported. | NQE reads Parquet, ORC, Apache Iceberg and Delta Lake. Hudi is covered by general Dataproc docs but not stated for NQE. Runs on x86 Intel and AMD, and ARM Google Axion C4A machines. |
| Where it runs | Official build and test platform is Ubuntu 20.04/22.04 or CentOS 8; the downloads page says binaries were built on CentOS 7 with static linking on Intel Xeon Platinum 8272CL class CPUs, and that performance and compatibility are not guaranteed on other OSes or hardware, so building from source is recommended. Docs have pages for S3, GCS and ABFS storage. | Google Cloud only. Rates on the pricing page are regional (us-central1 example given). |
| How open its benchmarks are | The repo includes the gluten-it integration test and benchmark tool (data generation, query runs, result comparison against a baseline) for building and running TPC-style queries locally, supported on Spark 3.4, 3.5, 4.0 and 4.1 profiles. The homepage TPC-H and TPC-DS numbers do not themselves link to a run configuration. | The launch blog footnotes the TPC-H-like derivation but publishes no scripts, cluster spec or raw timings. |
| Change Budget: code | No DataFrame or SQL changes. Gluten needs no DataFrame or SQL changes, only configuration. | No Spark code change stated. Google's documentation says Lightning Engine needs no Spark code changes. Spark, PySpark, SparkSQL and SparkR jobs are accelerated; other job types run on the standard engine. |
| Change Budget: cluster | Plugin, off-heap memory, shuffle manager. The settings are spark.plugins=org.apache.gluten.GlutenPlugin, off-heap memory enabled with a size, ColumnarShuffleManager and the bundle jar loaded ahead of Spark's jars. | Image 2.3.3+ or 3.0.2+, engine flag. Clusters need image 2.3.3 or later, or 3.0.2 or later (debian13 or ubuntu24), and the flag --engine=lightning at creation. NQE jobs on GPU accelerators fail. |
| Change Budget: cloud | None found on 2026-11-10. No cloud requirement is stated. Official build and test platforms are Ubuntu 20.04/22.04 or CentOS 8, and the docs have pages for S3, GCS and ABFS storage. | Google Cloud only. Google lists Lightning Engine on Managed Service for Apache Spark, with regional rates (us-central1 as the example). |
| Change Budget: vendor | Apache-2.0, no contract stated. Gluten's project license is Apache-2.0; no vendor or platform contract is described. | Premium tier or cluster add-on. Serverless premium is $0.089 per DCU-hour against $0.06 standard in us-central1. Clusters pay a management fee plus a Lightning Engine add-on plus Compute Engine VMs and disks. |
| Ledger entries | Gluten with Velox, TPC-DS 3 TB (project homepage) (run by its maker); Gluten with Velox, TPC-H 3 TB (project homepage) (run by its maker) | Lightning Engine against open source Spark, TPC-H-like 10 TB (run by its maker); Lightning Engine claim on Google's pricing page (run by its maker); Native Query Execution on Axion ARM, Google's documentation (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Apache Gluten with the Velox backend: gluten.apache.org, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/get-started/getting-st, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-b, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-sta, read 10 Nov 2026
- Apache Gluten with the Velox backend: gluten.apache.org/downloads, read 10 Nov 2026
- Apache Gluten with the Velox backend: raw.githubusercontent.com/apache/gluten/main/tools/gluten-it/README.md, read 10 Nov 2026
- Google Cloud Lightning Engine (Managed Service for Apache Spark): cloud.google.com/blog/products/data-analytics/introducing-lightning-engi, read 10 Nov 2026
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/pricing, read 10 Nov 2026
- Google Cloud Lightning Engine (Managed Service for Apache Spark): docs.cloud.google.com/managed-spark/docs/guides/lightning-engine, read 10 Nov 2026
- Apache Gluten Powered By page, read 10 Nov 2026
Last reviewed 10 Nov 2026.