Side by side, read 10 Nov 2026
Amazon EMR runtime for Apache Spark and Apache DataFusion Comet: the published facts in the same rows
Amazon EMR runtime for Apache Spark profileApache DataFusion Comet profile
The same rows for both
| Row | Amazon EMR runtime for Apache Spark | Apache DataFusion Comet |
|---|---|---|
| Kind | AWS-tuned Spark runtime on EMR | Open-source Spark plugin on DataFusion |
| Does well | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. AWS publishes aws-samples/emr-spark-benchmark, a TPC-DS harness for a 3 TB Parquet dataset with steps for OSS Spark, EMR on EC2 and EMR Serverless. | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. Releases have come roughly monthly since 1.0.0 on 2026-08-07, with 1.1.0 on 2026-10-06 and 1.2.0 on 2026-11-03. |
| Limited | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. AWS's up to 4.5x figure names no TPC-DS scale or cluster, and the benchmark repository gives commands and configs but no results. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. Comet supports Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later, and the project marks Delta Lake support as experimental. |
| Published speed results | AWS says the performance-optimized runtime executes up to 4.5x faster than open-source equivalents while delivering 2.7x faster Iceberg write performance. The page does not state the TPC-DS scale or cluster for these figures. | On an EKS cluster (r6i instances, EBS storage, data in S3), TPC-DS derived queries at scale factor 1000 (103 queries, zstd Parquet, each query run twice and the mean reported) took 994 s on Spark 4.2.0, 601 s on Comet 1.1.0 and 488 s on Comet 1.2.0. The page says Comet 1.2.0 is 2.0x faster than Spark and 1.2x faster than Comet 1.1.0. The same tables and settings on Spark 4.1.3: Spark 1,336 s, Comet 1.1.0 613 s (2.18x), Comet 1.2.0 532 s (2.51x). The project explains that Spark 4.1.3 is slower than 4.2.0 on this cluster mostly because of the default S3A input stream (classic vs S3 Analytics Accelerator), so the speedup depends on the Spark baseline; Spark 4.2.0 with the classic stream took 1,294 s. |
| Published cost results and prices | EMR on EC2 price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, from image download until pod end, rounded to the second, with a one-minute minimum, plus EKS and EC2 or Fargate cost. EMR Serverless is also available. Exact rates are region-specific and loaded by script on the page. | None found on 10 Nov 2026. |
| Code changes | Runtime is described as 100% API compatible with open source Spark, so Spark jobs need no code changes, but jobs must run on EMR (a migration if currently on other Spark hosts). | Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Unsupported operators and expressions fall back to Spark; the plan explain output marks fallback reasons with [COMET: ...]. Some expressions have native implementations that differ from Spark and are off unless a user sets allowIncompatible; Iceberg writes are off by default. |
| Tuning | The EMR docs list optimizations such as adaptive query execution, dynamic partition pruning, Bloom filter join and optimized join reorder. AQE-related features are on by default; some can be tuned with Spark properties. | The tuning guide says every deployment must configure how much memory Comet can use (off-heap pool and executor memory overhead), and has further pages for shuffle, Parquet and Iceberg scans, operators, and reducing row/columnar conversions. The benchmark config used spark.memory.offHeap.size=32G and spark.comet.exec.memoryPool.fraction=0.8 alongside 64G executor memory and 10G overhead. |
| Spark versions, platforms and table formats | EMR supports Apache Iceberg v3 and Spark 4.0 per the features page, and runs on EMR on EC2, EMR on EKS and EMR Serverless. | Comet supports Spark 3.5.9 (Scala 2.12/2.13), 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later (JDK 11 dropped in 1.1.0). The docs warn it may not fully work with cloud-provider Spark forks, so EMR, Databricks and similar managed runtimes are not documented as supported. Comet accelerates Iceberg tables with Parquet data files: a native Iceberg reader (on by default, tested with Iceberg 1.5, 1.8, 1.9, 1.10 and 1.11) and native writes (off by default via spark.comet.write.iceberg.enabled). Delta Lake support is marked experimental. The docs list known issues where results differ from Spark. Remote shuffle via Celeborn is supported. |
| Where it runs | AWS only. EMR runtime is available where EMR, EMR on EKS and EMR Serverless are available, and EMR on AWS Outposts is mentioned on the pricing page. | Published Maven Central jars bundle native libraries for Linux on amd64 and arm64 only; Apple Silicon macOS must build from source. Nightly snapshot jars are not Apache releases. The EKS benchmark ran on r6i instances; the EC2 guide recommends a c7i.4xlarge for single-node runs. No GPU or special hardware is required. |
| How open its benchmarks are | AWS publishes aws-samples/emr-spark-benchmark with steps to run TPC-DS on OSS Spark (EC2 via Flintrock), EMR on EC2, EMR Serverless and a link to EMR on EKS. It uses a 3 TB TPC-DS Parquet dataset in a public S3 bucket, 3 iterations, and 104 queries with a pre-built spark-benchmark-assembly-3.3.0.jar. The repo gives commands and configs; it does not state results. The repo README notes that on OSS Spark query q95 did not complete, so it was rerun alone and merged, and says this was not an issue on EMR. This shows the OSS baseline needed manual handling. A companion repo aws-samples/emr-on-eks-benchmark has the EMR on EKS TPC-DS benchmark with 3 TB examples and notes scale factor and executor config must change for 100 GB or 1 TB. | The project publishes benchmark scripts in benchmarks/tpc, data generation in the apache/datafusion-benchmarks repo, step-by-step guides for macOS, AWS EC2 (single node) and spark-sql-perf, and the full list of non-default Spark settings used for the TPC-DS run (executors 32 x 16 cores, 64G memory, shuffle partitions 512, S3A settings). Node count and instance size of the EKS cluster beyond the r6i family are not stated on the results page. |
| Change Budget: code | No code change stated. AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | No job code change stated. Comet is enabled with spark.plugins, a shuffle manager and off-heap settings, with no job code changes. Iceberg writes are off by default. |
| Change Budget: cluster | Run the jobs on an EMR release. Jobs must run on EMR (EC2, EKS or Serverless); a team on other Spark hosts migrates first. AQE-related features are on by default, and some can be tuned with Spark properties. | Plugin, shuffle manager, off-heap memory. Every deployment must configure how much memory Comet can use. Supported versions are Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with JDK 17 or later. |
| Change Budget: cloud | AWS only. AWS lists the runtime where EMR, EMR on EKS and EMR Serverless are available, and mentions EMR on AWS Outposts. | None found on 2026-11-10. No cloud requirement is stated. The published jars cover Linux on amd64 and arm64; the benchmark ran on EKS with r6i instances. |
| Change Budget: vendor | EMR charges added to EC2 and EBS. The EMR price is added to the EC2 and EBS price. EMR on EKS is charged per vCPU-hour and GB-hour requested, plus EKS and EC2 or Fargate cost. | Apache-2.0, no contract stated. Comet is Apache-2.0 licensed under the Apache Software Foundation; no vendor or platform contract is described. |
| Ledger entries | EMR runtime claim on the AWS features page (run by its maker) | Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 (run by its maker); Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 (run by its maker) |
How to read this page
The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns were run on different hardware, data and baselines unless the Ledger entry says otherwise, so the numbers are not comparable across columns. The Ledger method explains how entries are recorded.
Sources
- Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/features/spark, read 10 Nov 2026
- Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-spark-benchmark, read 10 Nov 2026
- Amazon EMR runtime for Apache Spark: github.com/aws-samples/emr-on-eks-benchmark, read 10 Nov 2026
- Amazon EMR runtime for Apache Spark: aws.amazon.com/emr/pricing, read 10 Nov 2026
- Amazon EMR runtime for Apache Spark: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/contr, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: raw.githubusercontent.com/apache/datafusion-comet/main/docs/source/user-, read 10 Nov 2026
- Apache DataFusion Comet: datafusion.apache.org/comet/contributor-guide/benchmarking.html, read 10 Nov 2026
- Apache DataFusion Comet GitHub repository, read 10 Nov 2026
Last reviewed 10 Nov 2026.