# What to check when AI training and feature pipelines wait on data preparation

Situation, read 10 Nov 2026

**The short answer.** Feature engineering and training-data preparation are Spark jobs with wide tables and many UDFs. Speedata, DataPelago and DualBird each name AI data preparation among their uses, with claims of 100x, 10X and 10x to 30x, none of which is tied to an AI workload on a public page.

## 5 things to check, with the published facts that answer each

### Does the vendor name AI data preparation, and is a result attached?

A use case on a home page is not a benchmark. Look for a named workload, data size and method.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [DataPelago](/options/datapelago/) | DataPelago's home page claims 10X faster at 80% cost reduction for GenAI and analytics data processing. No workload, baseline or method is attached. | [DataPelago home page](https://www.datapelago.ai/), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird lists AI data infrastructure (feature engineering and AI data prep) first among its use cases. Its one worked example with a published method is Iceberg compaction. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's published benchmark is TPC-DS at 3 TB with AWS (281 s on GPU). An AI data preparation benchmark: none found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | Speedata claims up to 100x for batch ETL and AI data preparation. A customer tile cites 62.7x on production Spark workloads for enterprise AI data preprocessing, with no customer name, data size or method. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Do your preparation jobs use UDFs, RDDs or custom types?

Python UDFs and RDD code are common in feature pipelines, and most accelerators leave them on the CPU.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Databricks Photon](/options/databricks-photon/) | Databricks says Photon does not support UDFs, RDD APIs or Dataset APIs. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | DataPelago lists heavy use of RDDs and UDFs among workloads to avoid. | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird says workloads dominated by Python UDFs fall back to the CPU and do not see the same benefit yet. | [DualBird post, A Simpler Spark](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Flarion](/options/flarion/) | Flarion says UDFs and UDAFs run on Spark as before. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google does not recommend Native Query Execution for RDD-heavy jobs, UDFs or most Spark ML libraries. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA says Dataset APIs with custom classes, flatMap, foreach, GraphX and RDDs are not accelerated. | [NVIDIA Spark plugin FAQ](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html), 10 Nov 2026 |
| [Speedata](/options/speedata/) | Speedata delegates UDFs to the CPU while the rest of the query runs on the APU. | [Speedata product page](https://www.speedata.io/product), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Which file and table formats hold your feature data?

Support differs by format and by read or write. Check the format your feature store writes.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Apache Auron](/options/apache-auron/) | Auron 8.0.0-incubating expands native scans for Iceberg, Hudi and Paimon. Delta Lake is not mentioned in the pages fetched. | [Apache Auron 8.0.0 release notes](https://auron.apache.org/archives/v8.0.0-incubating.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | Comet accelerates Iceberg tables with Parquet data files; native Iceberg writes are off by default and Delta Lake support is experimental. | [Comet Iceberg guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten's Velox backend fully supports Parquet scans and ORC partly; Iceberg is read-only and Hudi covers copy-on-write tables only. | [Gluten Velox backend limitations](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | Databricks says Photon accelerates Delta Lake, Iceberg and Parquet writes. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [Flarion](/options/flarion/) | Flarion lists Iceberg, Delta Lake and Hudi tables and Parquet, Avro and JSON files. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google's NQE reads Parquet, ORC, Iceberg and Delta Lake. Byte columns in ORC and Parquet and Struct, Array and Map in Parquet are unsupported inputs. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft lists Parquet, Delta and CSV, including data reached through OneLake shortcuts. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### What hardware does the accelerator add next to your training cluster?

Feature pipelines often run next to GPU training. Adding a second accelerator type changes scheduling and cost.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [DataPelago](/options/datapelago/) | DataPelago's Nucleus engine targets CPU, GPU and FPGA. The DPA-S docs pages read do not state a GPU requirement. | [DataPelago technology page](https://www.datapelago.ai/technology), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird runs on AWS F2 instances with up to 8 AMD Virtex UltraScale+ FPGAs each. | [AWS EC2 F2 instance page](https://aws.amazon.com/ec2/instance-types/f2/), 10 Nov 2026 |
| [Flarion](/options/flarion/) | Flarion runs CPU-native on instances already in use, and contrasts that with GPU accelerators. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's plugin needs GPUs from Volta to Blackwell, with driver R525 or later and Linux glibc 2.28 or later. | [NVIDIA cuDF plugin download page](https://nvidia.github.io/cudf-spark/docs/download.html), 10 Nov 2026 |
| [Speedata](/options/speedata/) | Speedata's C200 is a PCIe Gen 5 card, with a 2U server holding two cards. | [Speedata home page](https://www.speedata.io/), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### In which cloud does the data sit?

An option documented for one cloud is not an option for data in another.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS documents the runtime on EMR on EC2, EKS and Serverless. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | Databricks documents Photon on AWS, Azure and Google Cloud within Databricks compute. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | DataPelago documents AWS (EKS) and Google Cloud (Dataproc, GKE). | [DataPelago Accelerator for Spark page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird is AWS only; F2 is listed in eight AWS regions. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 |
| [Flarion](/options/flarion/) | Flarion lists AWS, Azure, GCP, on-premises and air-gapped use. | [Flarion FAQ](https://www.flarion.io/faqs), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents Lightning Engine on Google Cloud only. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft documents the engine only inside Fabric on Azure, on OneLake data. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

## Sources

- DataPelago home page: https://www.datapelago.ai/ (read 10 Nov 2026)
- DualBird home page: https://dualbird.io/ (read 10 Nov 2026)
- AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026)
- Speedata home page: https://www.speedata.io/ (read 10 Nov 2026)
- Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026)
- DataPelago Accelerator for Spark page: https://www.datapelago.ai/dpa (read 10 Nov 2026)
- DualBird post, A Simpler Spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026)
- Flarion FAQ: https://www.flarion.io/faqs (read 10 Nov 2026)
- Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)
- NVIDIA Spark plugin FAQ: https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html (read 10 Nov 2026)
- Speedata product page: https://www.speedata.io/product (read 10 Nov 2026)
- Apache Auron 8.0.0 release notes: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026)
- Comet Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Gluten Velox backend limitations: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/velox-backend-limitations.md (read 10 Nov 2026)
- Flarion product page: https://www.flarion.io/product (read 10 Nov 2026)
- Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026)
- DataPelago technology page: https://www.datapelago.ai/technology (read 10 Nov 2026)
- AWS EC2 F2 instance page: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026)
- NVIDIA cuDF plugin download page: https://nvidia.github.io/cudf-spark/docs/download.html (read 10 Nov 2026)
- Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026)
- AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026)
- Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/ai-data-prep/
