Skip to content
Spark Acceleration Compare

Situation, read 10 Nov 2026

What to check when AI training and feature pipelines wait on data preparation

All situationsOpen the Ledger

5 things to check, with the published facts that answer each

Does the vendor name AI data preparation, and is a result attached?

A use case on a home page is not a benchmark. Look for a named workload, data size and method.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
DataPelagoDataPelago's home page claims 10X faster at 80% cost reduction for GenAI and analytics data processing. No workload, baseline or method is attached.DataPelago home page, 10 Nov 2026
DualBirdDualBird lists AI data infrastructure (feature engineering and AI data prep) first among its use cases. Its one worked example with a published method is Iceberg compaction.DualBird home page, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's published benchmark is TPC-DS at 3 TB with AWS (281 s on GPU). An AI data preparation benchmark: none found on 2026-11-10.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataSpeedata claims up to 100x for batch ETL and AI data preparation. A customer tile cites 62.7x on production Spark workloads for enterprise AI data preprocessing, with no customer name, data size or method.Speedata home page, 10 Nov 2026

Do your preparation jobs use UDFs, RDDs or custom types?

Python UDFs and RDD code are common in feature pipelines, and most accelerators leave them on the CPU.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Databricks PhotonDatabricks says Photon does not support UDFs, RDD APIs or Dataset APIs.Databricks Photon documentation, 10 Nov 2026
DataPelagoDataPelago lists heavy use of RDDs and UDFs among workloads to avoid.DataPelago Accelerator for Spark page, 10 Nov 2026
DualBirdDualBird says workloads dominated by Python UDFs fall back to the CPU and do not see the same benefit yet.DualBird post, A Simpler Spark, 10 Nov 2026
FlarionFlarion says UDFs and UDAFs run on Spark as before.Flarion FAQ, 10 Nov 2026
Google Cloud Lightning EngineGoogle does not recommend Native Query Execution for RDD-heavy jobs, UDFs or most Spark ML libraries.Google Lightning Engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA says Dataset APIs with custom classes, flatMap, foreach, GraphX and RDDs are not accelerated.NVIDIA Spark plugin FAQ, 10 Nov 2026
SpeedataSpeedata delegates UDFs to the CPU while the rest of the query runs on the APU.Speedata product page, 10 Nov 2026

Which file and table formats hold your feature data?

Support differs by format and by read or write. Check the format your feature store writes.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Apache AuronAuron 8.0.0-incubating expands native scans for Iceberg, Hudi and Paimon. Delta Lake is not mentioned in the pages fetched.Apache Auron 8.0.0 release notes, 10 Nov 2026
Apache DataFusion CometComet accelerates Iceberg tables with Parquet data files; native Iceberg writes are off by default and Delta Lake support is experimental.Comet Iceberg guide, 10 Nov 2026
Apache Gluten with VeloxGluten's Velox backend fully supports Parquet scans and ORC partly; Iceberg is read-only and Hudi covers copy-on-write tables only.Gluten Velox backend limitations, 10 Nov 2026
Databricks PhotonDatabricks says Photon accelerates Delta Lake, Iceberg and Parquet writes.Databricks Photon documentation, 10 Nov 2026
FlarionFlarion lists Iceberg, Delta Lake and Hudi tables and Parquet, Avro and JSON files.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineGoogle's NQE reads Parquet, ORC, Iceberg and Delta Lake. Byte columns in ORC and Parquet and Struct, Array and Map in Parquet are unsupported inputs.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineMicrosoft lists Parquet, Delta and CSV, including data reached through OneLake shortcuts.Fabric native execution engine documentation, 10 Nov 2026

What hardware does the accelerator add next to your training cluster?

Feature pipelines often run next to GPU training. Adding a second accelerator type changes scheduling and cost.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
DataPelagoDataPelago's Nucleus engine targets CPU, GPU and FPGA. The DPA-S docs pages read do not state a GPU requirement.DataPelago technology page, 10 Nov 2026
DualBirdDualBird runs on AWS F2 instances with up to 8 AMD Virtex UltraScale+ FPGAs each.AWS EC2 F2 instance page, 10 Nov 2026
FlarionFlarion runs CPU-native on instances already in use, and contrasts that with GPU accelerators.Flarion FAQ, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's plugin needs GPUs from Volta to Blackwell, with driver R525 or later and Linux glibc 2.28 or later.NVIDIA cuDF plugin download page, 10 Nov 2026
SpeedataSpeedata's C200 is a PCIe Gen 5 card, with a 2U server holding two cards.Speedata home page, 10 Nov 2026

In which cloud does the data sit?

An option documented for one cloud is not an option for data in another.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS documents the runtime on EMR on EC2, EKS and Serverless.Amazon EMR pricing, 10 Nov 2026
Databricks PhotonDatabricks documents Photon on AWS, Azure and Google Cloud within Databricks compute.Databricks Photon documentation, 10 Nov 2026
DataPelagoDataPelago documents AWS (EKS) and Google Cloud (Dataproc, GKE).DataPelago Accelerator for Spark page, 10 Nov 2026
DualBirdDualBird is AWS only; F2 is listed in eight AWS regions.AWS EC2 instance types by Region, 10 Nov 2026
FlarionFlarion lists AWS, Azure, GCP, on-premises and air-gapped use.Flarion FAQ, 10 Nov 2026
Google Cloud Lightning EngineGoogle documents Lightning Engine on Google Cloud only.Google Managed Service for Apache Spark pricing, 10 Nov 2026
Microsoft Fabric native execution engineMicrosoft documents the engine only inside Fabric on Azure, on OneLake data.Fabric native execution engine documentation, 10 Nov 2026

Sources

  1. DataPelago home page, read 10 Nov 2026
  2. DualBird home page, read 10 Nov 2026
  3. AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
  4. Speedata home page, read 10 Nov 2026
  5. Databricks Photon documentation, read 10 Nov 2026
  6. DataPelago Accelerator for Spark page, read 10 Nov 2026
  7. DualBird post, A Simpler Spark, read 10 Nov 2026
  8. Flarion FAQ, read 10 Nov 2026
  9. Google Lightning Engine documentation, read 10 Nov 2026
  10. NVIDIA Spark plugin FAQ, read 10 Nov 2026
  11. Speedata product page, read 10 Nov 2026
  12. Apache Auron 8.0.0 release notes, read 10 Nov 2026
  13. Comet Iceberg guide, read 10 Nov 2026
  14. Gluten Velox backend limitations, read 10 Nov 2026
  15. Flarion product page, read 10 Nov 2026
  16. Fabric native execution engine documentation, read 10 Nov 2026
  17. DataPelago technology page, read 10 Nov 2026
  18. AWS EC2 F2 instance page, read 10 Nov 2026
  19. NVIDIA cuDF plugin download page, read 10 Nov 2026
  20. Amazon EMR pricing, read 10 Nov 2026
  21. AWS EC2 instance types by Region, read 10 Nov 2026
  22. Google Managed Service for Apache Spark pricing, read 10 Nov 2026

Last reviewed 10 Nov 2026.