Spark Adaptive Query Execution (AQE)

06.12.2023

Adaptive Query Execution (AQE) reoptimizes and adjusts query plans based on runtime metrics collected during the execution of the query. As of Spark 3.0, there are three major features in AQE: coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and optimizing skew join.

Summary Summary

Resources

Performance Tuning - Spark 3.5.0 Documentation, Apache Spark
Adaptive Query Execution: Speeding Up Spark SQL at Runtime, Databricks
Adaptive Query Execution Demo, Databricks

Spark Adaptive Query Execution (AQE) ​

Spark Adaptive Query Execution (AQE)