Situation, read 10 Nov 2026
What to check when AI training and feature pipelines wait on data preparation
5 things to check, with the published facts that answer each
Does the vendor name AI data preparation, and is a result attached?
A use case on a home page is not a benchmark. Look for a named workload, data size and method.
| Option | Published fact | Source, read |
|---|---|---|
| DataPelago | DataPelago's home page claims 10X faster at 80% cost reduction for GenAI and analytics data processing. No workload, baseline or method is attached. | DataPelago home page, 10 Nov 2026 |
| DualBird | DualBird lists AI data infrastructure (feature engineering and AI data prep) first among its use cases. Its one worked example with a published method is Iceberg compaction. | DualBird home page, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA's published benchmark is TPC-DS at 3 TB with AWS (281 s on GPU). An AI data preparation benchmark: none found on 2026-11-10. | AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026 |
| Speedata | Speedata claims up to 100x for batch ETL and AI data preparation. A customer tile cites 62.7x on production Spark workloads for enterprise AI data preprocessing, with no customer name, data size or method. | Speedata home page, 10 Nov 2026 |
Do your preparation jobs use UDFs, RDDs or custom types?
Python UDFs and RDD code are common in feature pipelines, and most accelerators leave them on the CPU.
| Option | Published fact | Source, read |
|---|---|---|
| Databricks Photon | Databricks says Photon does not support UDFs, RDD APIs or Dataset APIs. | Databricks Photon documentation, 10 Nov 2026 |
| DataPelago | DataPelago lists heavy use of RDDs and UDFs among workloads to avoid. | DataPelago Accelerator for Spark page, 10 Nov 2026 |
| DualBird | DualBird says workloads dominated by Python UDFs fall back to the CPU and do not see the same benefit yet. | DualBird post, A Simpler Spark, 10 Nov 2026 |
| Flarion | Flarion says UDFs and UDAFs run on Spark as before. | Flarion FAQ, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google does not recommend Native Query Execution for RDD-heavy jobs, UDFs or most Spark ML libraries. | Google Lightning Engine documentation, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA says Dataset APIs with custom classes, flatMap, foreach, GraphX and RDDs are not accelerated. | NVIDIA Spark plugin FAQ, 10 Nov 2026 |
| Speedata | Speedata delegates UDFs to the CPU while the rest of the query runs on the APU. | Speedata product page, 10 Nov 2026 |
Which file and table formats hold your feature data?
Support differs by format and by read or write. Check the format your feature store writes.
| Option | Published fact | Source, read |
|---|---|---|
| Apache Auron | Auron 8.0.0-incubating expands native scans for Iceberg, Hudi and Paimon. Delta Lake is not mentioned in the pages fetched. | Apache Auron 8.0.0 release notes, 10 Nov 2026 |
| Apache DataFusion Comet | Comet accelerates Iceberg tables with Parquet data files; native Iceberg writes are off by default and Delta Lake support is experimental. | Comet Iceberg guide, 10 Nov 2026 |
| Apache Gluten with Velox | Gluten's Velox backend fully supports Parquet scans and ORC partly; Iceberg is read-only and Hudi covers copy-on-write tables only. | Gluten Velox backend limitations, 10 Nov 2026 |
| Databricks Photon | Databricks says Photon accelerates Delta Lake, Iceberg and Parquet writes. | Databricks Photon documentation, 10 Nov 2026 |
| Flarion | Flarion lists Iceberg, Delta Lake and Hudi tables and Parquet, Avro and JSON files. | Flarion product page, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google's NQE reads Parquet, ORC, Iceberg and Delta Lake. Byte columns in ORC and Parquet and Struct, Array and Map in Parquet are unsupported inputs. | Google Lightning Engine documentation, 10 Nov 2026 |
| Microsoft Fabric native execution engine | Microsoft lists Parquet, Delta and CSV, including data reached through OneLake shortcuts. | Fabric native execution engine documentation, 10 Nov 2026 |
What hardware does the accelerator add next to your training cluster?
Feature pipelines often run next to GPU training. Adding a second accelerator type changes scheduling and cost.
| Option | Published fact | Source, read |
|---|---|---|
| DataPelago | DataPelago's Nucleus engine targets CPU, GPU and FPGA. The DPA-S docs pages read do not state a GPU requirement. | DataPelago technology page, 10 Nov 2026 |
| DualBird | DualBird runs on AWS F2 instances with up to 8 AMD Virtex UltraScale+ FPGAs each. | AWS EC2 F2 instance page, 10 Nov 2026 |
| Flarion | Flarion runs CPU-native on instances already in use, and contrasts that with GPU accelerators. | Flarion FAQ, 10 Nov 2026 |
| NVIDIA cuDF plugin for Apache Spark | NVIDIA's plugin needs GPUs from Volta to Blackwell, with driver R525 or later and Linux glibc 2.28 or later. | NVIDIA cuDF plugin download page, 10 Nov 2026 |
| Speedata | Speedata's C200 is a PCIe Gen 5 card, with a 2U server holding two cards. | Speedata home page, 10 Nov 2026 |
In which cloud does the data sit?
An option documented for one cloud is not an option for data in another.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS documents the runtime on EMR on EC2, EKS and Serverless. | Amazon EMR pricing, 10 Nov 2026 |
| Databricks Photon | Databricks documents Photon on AWS, Azure and Google Cloud within Databricks compute. | Databricks Photon documentation, 10 Nov 2026 |
| DataPelago | DataPelago documents AWS (EKS) and Google Cloud (Dataproc, GKE). | DataPelago Accelerator for Spark page, 10 Nov 2026 |
| DualBird | DualBird is AWS only; F2 is listed in eight AWS regions. | AWS EC2 instance types by Region, 10 Nov 2026 |
| Flarion | Flarion lists AWS, Azure, GCP, on-premises and air-gapped use. | Flarion FAQ, 10 Nov 2026 |
| Google Cloud Lightning Engine | Google documents Lightning Engine on Google Cloud only. | Google Managed Service for Apache Spark pricing, 10 Nov 2026 |
| Microsoft Fabric native execution engine | Microsoft documents the engine only inside Fabric on Azure, on OneLake data. | Fabric native execution engine documentation, 10 Nov 2026 |
Sources
- DataPelago home page, read 10 Nov 2026
- DualBird home page, read 10 Nov 2026
- AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
- Speedata home page, read 10 Nov 2026
- Databricks Photon documentation, read 10 Nov 2026
- DataPelago Accelerator for Spark page, read 10 Nov 2026
- DualBird post, A Simpler Spark, read 10 Nov 2026
- Flarion FAQ, read 10 Nov 2026
- Google Lightning Engine documentation, read 10 Nov 2026
- NVIDIA Spark plugin FAQ, read 10 Nov 2026
- Speedata product page, read 10 Nov 2026
- Apache Auron 8.0.0 release notes, read 10 Nov 2026
- Comet Iceberg guide, read 10 Nov 2026
- Gluten Velox backend limitations, read 10 Nov 2026
- Flarion product page, read 10 Nov 2026
- Fabric native execution engine documentation, read 10 Nov 2026
- DataPelago technology page, read 10 Nov 2026
- AWS EC2 F2 instance page, read 10 Nov 2026
- NVIDIA cuDF plugin download page, read 10 Nov 2026
- Amazon EMR pricing, read 10 Nov 2026
- AWS EC2 instance types by Region, read 10 Nov 2026
- Google Managed Service for Apache Spark pricing, read 10 Nov 2026
Last reviewed 10 Nov 2026.