# What to check when daily Iceberg compaction is a large part of the Spark bill

Situation, read 10 Nov 2026

**The short answer.** A team paying for daily Iceberg compaction should check who ran any benchmark, whether its table matches theirs, what cost per terabyte includes, and where the job can run. Of the twelve options, one has a compaction benchmark: DualBird's, run by DualBird, method behind a sign-up form (checked 2026-11-10, the read date).

## 5 things to check, with the published facts that answer each

### Has anyone published a compaction benchmark, and who ran it?

A compaction result run by the seller and described without scripts is a claim to verify.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | None found on 2026-11-10. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 |
| [Apache Auron](/options/apache-auron/) | An Auron compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS at 1 TB. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | A Comet compaction benchmark: none found on 2026-11-10. Its published benchmark is TPC-DS. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird's own figure, from its post of June 11, 2026 (the post date; 2026-11-10 is only the day it was read): sort-based Iceberg compaction on 100 GB, about 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark. DualBird ran it. The white paper's method is behind a form, and no public repository was found. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Which options read and write Iceberg through the accelerator?

Compaction is a write job. An accelerator that reads Iceberg but writes through vanilla Spark leaves the expensive step alone.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS lists Apache Iceberg v3 support and claims 2.7x faster Iceberg write performance, with no workload stated. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 |
| [Apache Auron](/options/apache-auron/) | Auron 8.0.0-incubating added native scan support for Iceberg, Hudi and Paimon, including copy-on-write tables and Iceberg changelog scans. | [Apache Auron 8.0.0 release notes](https://auron.apache.org/archives/v8.0.0-incubating.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | Comet has a native Iceberg reader on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11 on Parquet data files. Native writes are off by default (spark.comet.write.iceberg.enabled). | [Comet Iceberg guide](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | Gluten's Iceberg page says only reading is supported, and only Spark 3.4 is well tested. | [Gluten Velox Iceberg page](https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | Databricks says Photon accelerates Delta Lake, Apache Iceberg and Parquet writes. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | Iceberg appears in DualBird's compaction benchmark and in the meta description of its December 2025 demo post. A table format support list: none found on 2026-11-10. | [DualBird home page](https://dualbird.io/?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Flarion](/options/flarion/) | Flarion lists Iceberg, Delta Lake and Hudi table formats. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google's Native Query Execution reads Parquet, ORC, Apache Iceberg and Delta Lake. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's user guide has Iceberg and Delta Lake support pages, including reading and writing tables. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### What does compaction cost per terabyte, and what is included?

Cost per terabyte is the number that compares to your current bill, if the instance, storage and license costs are all in it.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS publishes no cost figure for the Iceberg write claim: none found on 2026-11-10. EMR price is added to EC2 and EBS cost. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 |
| [Apache Auron](/options/apache-auron/) | None found on 2026-11-10. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | A cost-per-job result for Comet: none found on 2026-11-10. | [Comet benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | A cost-per-job result for Gluten: none found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird's own figure, baseline instance types not stated: about $2 per TB compacted against about $6 per TB rising past $13 for CPU clusters. DualBird attributes part of the saving to F2 local NVMe replacing EBS volumes. The plugin's pricing model is not stated. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Does the test table look like yours?

Table width, row count, compression and sort order are the inputs a compaction result depends on, so match them before trusting a ratio.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | None found on 2026-11-10. | [Amazon EMR Spark features page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 |
| [Apache Auron](/options/apache-auron/) | Auron's TPC-DS 1 TB data is Parquet with zstd; node types and counts are not given. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | Comet's Parquet data in its TPC-DS run is zstd compressed at scale factor 1000, on r6i instances with EBS. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | DualBird's own test dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3, 100 GB. Instance types and counts, and Spark version, are not stated in the public post. | [DualBird Iceberg compaction benchmark post](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=sparkaccelerationcompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | None found on 2026-11-10. | [Google Lightning Engine documentation](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | None found on 2026-11-10. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | None found on 2026-11-10. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Where can the compaction job run?

A daily job needs capacity every day, in the region and cloud that holds the tables.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR runtime for Apache Spark](/options/emr-spark-runtime/) | AWS documents the EMR runtime on EMR on EC2, EMR on EKS and EMR Serverless. | [Amazon EMR pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 |
| [Apache Auron](/options/apache-auron/) | None found on 2026-11-10. | [Apache Auron benchmark page](https://auron.apache.org/documents/benchmarks.html), 10 Nov 2026 |
| [Apache DataFusion Comet](/options/datafusion-comet/) | None found on 2026-11-10. | [Comet TPC-DS benchmark results](https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md), 10 Nov 2026 |
| [Apache Gluten with Velox](/options/apache-gluten-velox/) | None found on 2026-11-10. | [Apache Gluten homepage](https://gluten.apache.org/), 10 Nov 2026 |
| [Databricks Photon](/options/databricks-photon/) | None found on 2026-11-10. | [Databricks Photon documentation](https://docs.databricks.com/en/compute/photon.html), 10 Nov 2026 |
| [DataPelago](/options/datapelago/) | None found on 2026-11-10. | [DataPelago DPA page](https://www.datapelago.ai/dpa), 10 Nov 2026 |
| [DualBird](/options/dualbird/) | F2 instances are listed in us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2. DualBird is AWS only. | [AWS EC2 instance types by Region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 |
| [Flarion](/options/flarion/) | None found on 2026-11-10. | [Flarion product page](https://www.flarion.io/product), 10 Nov 2026 |
| [Google Cloud Lightning Engine](/options/dataproc-lightning-engine/) | Google documents Lightning Engine on Google Cloud only. | [Google Managed Service for Apache Spark pricing](https://docs.cloud.google.com/managed-spark/pricing), 10 Nov 2026 |
| [Microsoft Fabric native execution engine](/options/fabric-native-execution-engine/) | Microsoft documents the engine only inside Fabric on Azure. | [Fabric native execution engine documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview), 10 Nov 2026 |
| [NVIDIA cuDF plugin for Apache Spark](/options/nvidia-rapids-spark/) | NVIDIA's AWS test ran in us-east-1 on g7.4xlarge instances. | [AWS Big Data blog on GPU Spark with EMR on EKS](https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/), 10 Nov 2026 |
| [Speedata](/options/speedata/) | None found on 2026-11-10. | [Speedata technology page](https://www.speedata.io/technology), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

## Sources

- Amazon EMR Spark features page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026)
- Apache Auron benchmark page: https://auron.apache.org/documents/benchmarks.html (read 10 Nov 2026)
- Comet TPC-DS benchmark results: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contributor-guide/benchmark-results/tpc-ds.md (read 10 Nov 2026)
- Apache Gluten homepage: https://gluten.apache.org/ (read 10 Nov 2026)
- DataPelago DPA page: https://www.datapelago.ai/dpa (read 10 Nov 2026)
- Databricks Photon documentation: https://docs.databricks.com/en/compute/photon.html (read 10 Nov 2026)
- DualBird Iceberg compaction benchmark post: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026)
- Flarion product page: https://www.flarion.io/product (read 10 Nov 2026)
- Google Lightning Engine documentation: https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine (read 10 Nov 2026)
- Fabric native execution engine documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview (read 10 Nov 2026)
- AWS Big Data blog on GPU Spark with EMR on EKS: https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/ (read 10 Nov 2026)
- Speedata technology page: https://www.speedata.io/technology (read 10 Nov 2026)
- Apache Auron 8.0.0 release notes: https://auron.apache.org/archives/v8.0.0-incubating.html (read 10 Nov 2026)
- Comet Iceberg guide: https://raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-guide/latest/iceberg.md (read 10 Nov 2026)
- Gluten Velox Iceberg page: https://raw.githubusercontent.com/apache/gluten/main/docs/backends/velox/get-started/VeloxIceberg.md (read 10 Nov 2026)
- DualBird home page: https://dualbird.io/ (read 10 Nov 2026)
- Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026)
- Comet benchmarking guide: https://datafusion.apache.org/comet/contributor-guide/benchmarking.html (read 10 Nov 2026)
- AWS EC2 instance types by Region: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026)
- Google Managed Service for Apache Spark pricing: https://docs.cloud.google.com/managed-spark/pricing (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://sparkaccelerationcompare.com/situations/iceberg-compaction/
