Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Amazon EMR runtime for Apache Spark and DataPelago: the published facts in the same rows

Amazon EMR runtime for Apache Spark profileDataPelago profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowAmazon EMR runtime for Apache SparkDataPelago
KindAWS-tuned Spark runtime on EMRGluten-based CPU and GPU accelerator
Does wellAWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless.Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge.
LimitedThe runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results.DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned.
Published speed resultsAWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures.The home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs.
Published cost results and pricesEMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page.Named customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs.
Code changesRuntime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts).DPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN.
TuningThe EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties.Mandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors.
Spark versions, platforms and table formatsEMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless.On AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed.
Where it runsAWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page.Runs in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed.
How open its benchmarks areAWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB.None found on 10 Nov 2026.
Change Budget: codeNo code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes.No application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs.
Change Budget: clusterRun the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties.Plugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS.
Change Budget: cloudAWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts.AWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account.
Change Budget: vendorEMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost.Marketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read.
Ledger entriesEMR runtime claim on the AWS features page (run by its maker)Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark, read 10 Nov 2026
  2. Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark, read 10 Nov 2026
  3. Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark, read 10 Nov 2026
  4. Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing, read 10 Nov 2026
  5. Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html, read 10 Nov 2026
  6. DataPelago: datapelago.ai, read 10 Nov 2026
  7. DataPelago: datapelago.ai/dpa, read 10 Nov 2026
  8. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html, read 10 Nov 2026
  9. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html, read 10 Nov 2026
  10. DataPelago: datapelago.ai/technology, read 10 Nov 2026
  11. DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF, read 10 Nov 2026

Last reviewed 10 Nov 2026.