Stage-Level Executor Allocation in Apache Spark™ with Cost–Performance Trade-offs

Allocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost guaranteed to be sub-optimal for each of the thousands of diverse, individual stages executed by the application. Users may also have explicit preferences, such as completing an application within a specific time bud- get while minimizing cost, that existing solutions usually fail to support. We propose a novel method for determining the number of executors per stage in a serverless Apache Spark™ environment, enabling users to specify their desired cost–performance trade-off. Our approach trains tree-ensemble models to estimate the run times and costs of a stage as a function of allocated resources. These estimates are then used to recommend resources for each stage individually. We evaluate our approach on TPC-DS and SQLStorm benchmarks and compare it against two baselines. Depending on the user-defined trade-off parameter and setup, our approach achieves ∼50% cost savings across 103 TPC-DS queries with only a ∼16% slow- down, and ∼40.5% on 96 SQLStorm queries at a ∼29% slowdown.