# Scout: An Experienced Guide to Find the Best Cloud Configuration

arXiv: https://arxiv.org/abs/1803.01296 · HTML: https://mlsanity.com/paper/1803.01296/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2018.

## Built on by (most cited first, 4 of 4)

- [Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts](https://mlsanity.com/paper/2107.13921/) (2021)
- [Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation](https://mlsanity.com/paper/2108.12211/) (2021)
- [Search-based Methods for Multi-Cloud Configuration](https://mlsanity.com/paper/2204.09437/) (2022)
- [Hierarchical Upper Confidence Bounds for Constrained Online Learning](https://mlsanity.com/paper/2410.17216/) (2024)

## Abstract (arXiv)

Finding the right cloud configuration for workloads is an essential step to ensure good performance and contain running costs. A poor choice of cloud configuration decreases application performance and increases running cost significantly. While Bayesian Optimization is effective and applicable to any workloads, it is fragile because performance and workload are hard to model (to predict). In this paper, we propose a novel method, SCOUT. The central insight of SCOUT is that using prior measurements, even those for different workloads, improves search performance and reduces search cost. At its core, SCOUT extracts search hints (inference of resource requirements) from low-level performance metrics. Such hints enable SCOUT to navigate through the search space more efficiently---only spotlight region will be searched. We evaluate SCOUT with 107 workloads on Apache Hadoop and Spark. The experimental results demonstrate that our approach finds better cloud configurations with a lower search cost than state of the art methods. Based on this work, we conclude that (i) low-level performance information is necessary for finding the right cloud configuration in an effective, efficient and reliable way, and (ii) a search method can be guided by historical data, thereby reducing cost and improving performance.