Skip to content
Spark Acceleration Compare

Situation, read 10 Nov 2026

What to check when daily Iceberg compaction is a large part of the Spark bill

All situationsOpen the Ledger

5 things to check, with the published facts that answer each

Has anyone published a compaction benchmark, and who ran it?

A compaction result run by the seller and described without scripts is a claim to verify.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkNone found on 2026-11-10.Amazon EMR Spark features page, 10 Nov 2026
Apache AuronAn Auron compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS at 1 TB.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometA Comet compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS.Comet TPC-DS benchmark results, 10 Nov 2026
Apache Gluten with VeloxNone found on 2026-11-10.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonNone found on 2026-11-10.Databricks Photon documentation, 10 Nov 2026
DataPelagoNone found on 2026-11-10.DataPelago DPA page, 10 Nov 2026
DualBirdDualBird's own figure, from its post of June 11, 2026 (the post date; 2026-11-10 is only the day it was read): sort-based Iceberg compaction on 100 GB, about 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark. DualBird ran it. The white paper's method is behind a form, and no public repository was found.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionNone found on 2026-11-10.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineNone found on 2026-11-10.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineNone found on 2026-11-10.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNone found on 2026-11-10.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataNone found on 2026-11-10.Speedata technology page, 10 Nov 2026

Which options read and write Iceberg through the accelerator?

Compaction is a write job. An accelerator that reads Iceberg but writes through vanilla Spark leaves the expensive step alone.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS lists Apache Iceberg v3 support and claims 2.7x faster Iceberg write performance, with no workload stated.Amazon EMR Spark features page, 10 Nov 2026
Apache AuronAuron 8.0.0-incubating added native scan support for Iceberg, Hudi and Paimon, including copy-on-write tables and Iceberg changelog scans.Apache Auron 8.0.0 release notes, 10 Nov 2026
Apache DataFusion CometComet has a native Iceberg reader on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11 on Parquet data files. Native writes are off by default (spark.comet.write.iceberg.enabled).Comet Iceberg guide, 10 Nov 2026
Apache Gluten with VeloxGluten's Iceberg page says only reading is supported, and only Spark 3.4 is well tested.Gluten Velox Iceberg page, 10 Nov 2026
Databricks PhotonDatabricks says Photon accelerates Delta Lake, Apache Iceberg and Parquet writes.Databricks Photon documentation, 10 Nov 2026
DataPelagoNone found on 2026-11-10.DataPelago DPA page, 10 Nov 2026
DualBirdIceberg appears in DualBird's compaction benchmark and in the meta description of its December 2025 demo post. A table format support list: none found on 2026-11-10.DualBird home page, 10 Nov 2026
FlarionFlarion lists Iceberg, Delta Lake and Hudi table formats.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineGoogle's Native Query Execution reads Parquet, ORC, Apache Iceberg and Delta Lake.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineNone found on 2026-11-10.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's user guide has Iceberg and Delta Lake support pages, including reading and writing tables.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataNone found on 2026-11-10.Speedata technology page, 10 Nov 2026

What does compaction cost per terabyte, and what is included?

Cost per terabyte is the number that compares to your current bill, if the instance, storage and license costs are all in it.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS publishes no cost figure for the Iceberg write claim: none found on 2026-11-10. EMR price is added to EC2 and EBS cost.Amazon EMR pricing, 10 Nov 2026
Apache AuronNone found on 2026-11-10.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometA cost-per-job result for Comet: none found on 2026-11-10.Comet benchmarking guide, 10 Nov 2026
Apache Gluten with VeloxA cost-per-job result for Gluten: none found on 2026-11-10.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonNone found on 2026-11-10.Databricks Photon documentation, 10 Nov 2026
DataPelagoNone found on 2026-11-10.DataPelago DPA page, 10 Nov 2026
DualBirdDualBird's own figure, baseline instance types not stated: about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. DualBird attributes part of the saving to F2 local NVMe replacing EBS volumes. The plugin's pricing model is not stated.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionNone found on 2026-11-10.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineNone found on 2026-11-10.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineNone found on 2026-11-10.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNone found on 2026-11-10.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataNone found on 2026-11-10.Speedata technology page, 10 Nov 2026

Does the test table look like yours?

Table width, row count, compression and sort order are the inputs a compaction result depends on, so match them before trusting a ratio.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkNone found on 2026-11-10.Amazon EMR Spark features page, 10 Nov 2026
Apache AuronAuron's TPC-DS 1 TB data is Parquet with zstd; node types and counts are not given.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometComet's Parquet data in its TPC-DS run is zstd compressed at scale factor 1000, on r6i instances with EBS.Comet TPC-DS benchmark results, 10 Nov 2026
Apache Gluten with VeloxNone found on 2026-11-10.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonNone found on 2026-11-10.Databricks Photon documentation, 10 Nov 2026
DataPelagoNone found on 2026-11-10.DataPelago DPA page, 10 Nov 2026
DualBirdDualBird's own test dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3, 100 GB. Instance types and counts, and Spark version, are not stated in the public post.DualBird Iceberg compaction benchmark post, 10 Nov 2026
FlarionNone found on 2026-11-10.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineNone found on 2026-11-10.Google Lightning Engine documentation, 10 Nov 2026
Microsoft Fabric native execution engineNone found on 2026-11-10.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNone found on 2026-11-10.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataNone found on 2026-11-10.Speedata technology page, 10 Nov 2026

Where can the compaction job run?

A daily job needs capacity every day, in the region and cloud that holds the tables.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR runtime for Apache SparkAWS documents the EMR runtime on EMR on EC2, EMR on EKS and EMR Serverless.Amazon EMR pricing, 10 Nov 2026
Apache AuronNone found on 2026-11-10.Apache Auron benchmark page, 10 Nov 2026
Apache DataFusion CometNone found on 2026-11-10.Comet TPC-DS benchmark results, 10 Nov 2026
Apache Gluten with VeloxNone found on 2026-11-10.Apache Gluten homepage, 10 Nov 2026
Databricks PhotonNone found on 2026-11-10.Databricks Photon documentation, 10 Nov 2026
DataPelagoNone found on 2026-11-10.DataPelago DPA page, 10 Nov 2026
DualBirdF2 instances are listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2. DualBird is AWS only.AWS EC2 instance types by Region, 10 Nov 2026
FlarionNone found on 2026-11-10.Flarion product page, 10 Nov 2026
Google Cloud Lightning EngineGoogle documents Lightning Engine on Google Cloud only.Google Managed Service for Apache Spark pricing, 10 Nov 2026
Microsoft Fabric native execution engineMicrosoft documents the engine only inside Fabric on Azure.Fabric native execution engine documentation, 10 Nov 2026
NVIDIA cuDF plugin for Apache SparkNVIDIA's AWS test ran in us-east-1 on g7.4xlarge instances.AWS Big Data blog on GPU Spark with EMR on EKS, 10 Nov 2026
SpeedataNone found on 2026-11-10.Speedata technology page, 10 Nov 2026

Sources

  1. Amazon EMR Spark features page, read 10 Nov 2026
  2. Apache Auron benchmark page, read 10 Nov 2026
  3. Comet TPC-DS benchmark results, read 10 Nov 2026
  4. Apache Gluten homepage, read 10 Nov 2026
  5. DataPelago DPA page, read 10 Nov 2026
  6. Databricks Photon documentation, read 10 Nov 2026
  7. DualBird Iceberg compaction benchmark post, read 10 Nov 2026
  8. Flarion product page, read 10 Nov 2026
  9. Google Lightning Engine documentation, read 10 Nov 2026
  10. Fabric native execution engine documentation, read 10 Nov 2026
  11. AWS Big Data blog on GPU Spark with EMR on EKS, read 10 Nov 2026
  12. Speedata technology page, read 10 Nov 2026
  13. Apache Auron 8.0.0 release notes, read 10 Nov 2026
  14. Comet Iceberg guide, read 10 Nov 2026
  15. Gluten Velox Iceberg page, read 10 Nov 2026
  16. DualBird home page, read 10 Nov 2026
  17. Amazon EMR pricing, read 10 Nov 2026
  18. Comet benchmarking guide, read 10 Nov 2026
  19. AWS EC2 instance types by Region, read 10 Nov 2026
  20. Google Managed Service for Apache Spark pricing, read 10 Nov 2026

Last reviewed 10 Nov 2026.