Skip to content
Spark Acceleration Compare

Option profile, read 10 Nov 2026

DataPelago, Gluten-based CPU and GPU accelerator: the published facts

DataPelago in the LedgerCompare with another option

Kind
Gluten-based CPU and GPU accelerator
Website
www.datapelago.ai
Name checked
DataPelago, Inc. (Mountain View, CA) is live. datapelago.com redirects to www.datapelago.ai; page links also reference datapelago.io. The Spark product is named DataPelago Accelerator for Spark (DPA-S), launched Aug 2025.
Read
10 Nov 2026

What it does well

  • Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE.
  • DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge.

Where it is limited

  • DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method.
  • Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read.
  • The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned.

Published speed results

FactSource typeSource, read
The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures.vendordatapelago.ai, 10 Nov 2026
DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs.vendordatapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF, 10 Nov 2026

Published cost results and prices

FactSource typeSource, read
Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method.vendordatapelago.ai, 10 Nov 2026
The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs.vendordatapelago.ai/dpa, 10 Nov 2026

Code changes

FactSource typeSource, read
DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN.vendordatapelago.ai/dpa, 10 Nov 2026

Tuning

FactSource typeSource, read
Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors.vendor docsdatapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html, 10 Nov 2026

Spark versions, platforms and table formats

FactSource typeSource, read
On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site.vendor docsdatapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html, 10 Nov 2026
GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7.vendor docsdatapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html, 10 Nov 2026
The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed.vendordatapelago.ai, 10 Nov 2026

Where it runs

FactSource typeSource, read
Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed.vendordatapelago.ai/technology, 10 Nov 2026

Profile

FactSource typeSource, read
Founder and CEO Rajan Goyal was previously CTO at Fungible. DataPelago, Inc. is at 100 View Street, Mountain View, CA. Timeline from the press page: Universal Data Processing Engine unveiled Oct 1, 2024; President John Chirapurath appointed Jul 28, 2025; DPA-S launched Aug 5, 2025; Nucleus-versus-cuDF release Aug 22, 2025; Fast Company Most Innovative Companies of 2026 (Mar 24, 2026); Data Breakthrough award (May 13, 2026). Investor/partner logos on the home page include Taiwania, Qualcomm, Eclipse, Alter and Nautilus (alt text); no round sizes were found.vendordatapelago.ai/press, 10 Nov 2026

Ledger entries for DataPelago

PublishedOptionTestResult as publishedRan byMethod published
2025-08-21DataPelagoNucleus v3.0 against NVIDIA cuDF 25.04.00, operator testsUp to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDFDataPelago (its maker)No

Change Budget row

ColumnWhat changesDetailSource, read
CodeNo application rewrite statedDataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs.DataPelago Accelerator for Spark page, 10 Nov 2026
ClusterPlugin, license and memory settingsMandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS.DataPelago AWS deployment docs, 10 Nov 2026
CloudAWS EKS or Google CloudDataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account.DataPelago Google Cloud deployment docs, 10 Nov 2026
VendorMarketplace subscription requiredDPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read.DataPelago AWS deployment docs, 10 Nov 2026

Searched for and none found on 10 Nov 2026

  • TPC-DS/TPC-H or end-to-end Spark benchmark with published method, and any third-party benchmark (searched: datapelago.ai, datapelago.ai/dpa, datapelago.ai/technology)
  • Funding amounts, round dates and lead investors (searched: datapelago.ai/company, datapelago.ai/press, datapelago.ai)
  • Published unit prices (searched: datapelago.ai/dpa, datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html)

DataPelago side by side with each other option

Sources

  1. DataPelago: datapelago.ai, read 10 Nov 2026
  2. DataPelago: datapelago.ai/dpa, read 10 Nov 2026
  3. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html, read 10 Nov 2026
  4. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html, read 10 Nov 2026
  5. DataPelago: datapelago.ai/technology, read 10 Nov 2026
  6. DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF, read 10 Nov 2026
  7. DataPelago: datapelago.ai/press, read 10 Nov 2026

Last reviewed 10 Nov 2026.