Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

DataPelago and DualBird: the published facts in the same rows

DataPelago profileDualBird profile

The same rows for both

Options in alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026.
RowDataPelagoDualBird
KindGluten-based CPU and GPU acceleratorFPGA Spark plugin on AWS F2
Does wellPer DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. DataPelago's blog publishes operator microbenchmarks of Nucleus v3.0 against NVIDIA cuDF 25.04.00 on one H100 GPU of an AWS p5.48xlarge.DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). DualBird says onboarding is an instance type change to F2 plus its Spark plugin, and that Spark still handles planning, scheduling and fault tolerance.
LimitedDataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. Pricing is pay-as-you-go through the AWS and GCP marketplaces, and no unit price is published on the pages read. The AWS docs describe EKS with Spark 3.5.1 only, and EMR and Databricks are not mentioned.DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. F2 is listed in eight AWS regions (not, for example, us-east-2 or eu-west-1), the method for its compaction benchmark is behind a sign-up form, and no price for the plugin is published. A DualBird engineer states that workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet.
Published speed resultsThe home page headline claims 10X faster at 80% cost reduction for GenAI and analytics data processing. The DPA-S page says it can reduce costs by up to 80% on qualifying workloads without rewriting applications or migrating data. No workload, baseline or method is attached to the 10X/80% figures. DataPelago's blog (Aug 21, 2025) compares Nucleus v3.0 with NVIDIA cuDF 25.04.00 on one H100 of an AWS p5.48xlarge, for single operators. Nucleus is up to 10.5x faster on project, up to 10.1x on filter, up to 4.3x on aggregate, up to 4.5x on hash aggregate, and up to 8.2x on Top-K. These are operator microbenchmarks against a GPU library, not end-to-end Spark jobs.The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages.
Published cost results and pricesNamed customer quotes on the vendor site: Akad Seguros (CTO) says costs fell by more than 50%; ShareChat (VP of Data) says the heavy OLAP cube jobs now complete on OSS Spark and costs fell by 50%; RevSure.ai (CEO) reports gains in days with zero application changes; McAfee and Samsung SDS America describe 'promising results' only. These are testimonials with no baseline, workload or method. The DPA-S page describes pay-as-you-go pricing billed through the AWS and GCP marketplaces ('Use more and Save more'). No price per unit is published on the pages read. The docs require a Subscription ID and license bucket path tied to registered AWS account IDs.No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings.
Code changesDPA-S is presented as working with existing Spark clusters and jobs with no application rewrites or data migration; languages are PySpark, Scala and Spark SQL. Workloads to select: compute-intensive Parquet/ORC SQL and DataFrame jobs. Workloads to avoid: heavy use of RDDs and UDFs. The docs list limitations: native CPU acceleration is not used when ANSI mode is on (queries run on the JVM), case-sensitive mode is not supported and may return incorrect results, plus listed caveats for regexp functions, file source formats, JSON functions and NaN.Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding.
TuningMandatory Spark settings in the EKS docs include spark.plugins=org.apache.gluten.GlutenPlugin, the subscription ID and license location env vars, spark.memory.offHeap.enabled=true with a positive spark.memory.offHeap.size, spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager and netty Java options. CBO settings are recommended. The DPA-S page tells users to adjust Spark configuration (shuffle manager, memory tuning). So some per-deployment configuration is required, in contrast to 'no tuning' claims by other vendors.DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting.
Spark versions, platforms and table formatsOn AWS, DPA-S is available for OSS Spark on EKS as a Docker container in ECR: Spark 3.5.1, Java 11, Scala 2.12, Ubuntu 22. EMR and Databricks are not mentioned on the AWS docs page. The vendor page says 'multiple Spark versions on Google Cloud and AWS'. Lakehouse formats Iceberg, Hudi and Delta are named only in a reseller testimonial (Twingo) on the vendor site. GCP docs list supported combinations: Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, OSS Spark 3.5.1 and OSS Spark 3.3.4 (all Java 11, Scala 2.12) and a GKE deployment. Docs version header is 3.1.7. The technology page says the engine builds on Substrait-based open-source frameworks such as Apache Gluten and Velox; the docs configure the Gluten plugin class. The Gluten initiator is quoted on the home page in support. Nucleus and DPA-S themselves are proprietary and licensed.The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg').
Where it runsRuns in the customer's own cloud account on AWS (EKS) and Google Cloud (Dataproc, GKE). The Nucleus engine targets CPU, GPU and FPGA. The Nucleus-versus-cuDF test used an AWS p5.48xlarge with H100 GPUs; the DPA-S docs pages read do not state a GPU requirement, so the hardware needed for DPA-S itself was not confirmed.DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'.
How open its benchmarks areNone found on 10 Nov 2026.The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404).
Change Budget: codeNo application rewrite stated. DataPelago presents DPA-S as working with existing Spark clusters and jobs with no application rewrites or data migration. It tells users to avoid heavy use of RDDs and UDFs.No code change stated. DualBird's about page says 'No infrastructure changes. No code changes.'
Change Budget: clusterPlugin, license and memory settings. Mandatory settings include the Gluten plugin class, the subscription ID and license location, off-heap memory, the columnar shuffle manager and Netty Java options. On AWS it ships as a Docker container in ECR for Spark 3.5.1 on EKS.Instance type change to F2. DualBird's onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. Cluster sizing then reduces to the number of F2 nodes, per DualBird.
Change Budget: cloudAWS EKS or Google Cloud. DataPelago documents EKS on AWS and Dataproc and GKE on Google Cloud, in the customer's own account.AWS only, F2 in eight regions. F2 is listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2, and not, for example, in us-east-2 or eu-west-1. DualBird mentions no other cloud.
Change Budget: vendorMarketplace subscription required. DPA-S is billed pay-as-you-go through the AWS and GCP marketplaces and needs a Subscription ID and license bucket tied to registered AWS account IDs. No unit price is published on the pages read.Plugin contract; no price published. No pricing, price list or marketplace listing was found on DualBird's site, and the pricing model for the plugin is not stated.
Ledger entriesNucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests (run by its maker)DualBird sort-based Iceberg compaction, 100 GB (run by its maker); DualBird 110 GB enrichment pipeline (join denormalization) (run by its maker); DualBird single task performance tile (run by its maker)

How to read this page

The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.

Sources

  1. DataPelago: datapelago.ai, read 10 Nov 2026
  2. DataPelago: datapelago.ai/dpa, read 10 Nov 2026
  3. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-aws.html, read 10 Nov 2026
  4. DataPelago: datapelago-docs.readthedocs-hosted.com/en/latest/deploy-on-gcp.html, read 10 Nov 2026
  5. DataPelago: datapelago.ai/technology, read 10 Nov 2026
  6. DataPelago: datapelago.ai/resources/DataPelago-Nucleus-Vs-Nvidia-cuDF, read 10 Nov 2026
  7. DualBird: dualbird.io, read 10 Nov 2026
  8. DualBird: dualbird.io/product, read 10 Nov 2026
  9. DualBird: dualbird.io/resources/blog/a-simpler-spark, read 10 Nov 2026
  10. DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low, read 10 Nov 2026
  11. DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper, read 10 Nov 2026
  12. DualBird: aws.amazon.com/ec2/instance-types/f2, read 10 Nov 2026
  13. DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html, read 10 Nov 2026
  14. DualBird about page, read 10 Nov 2026

Last reviewed 10 Nov 2026.