# Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics

arXiv: https://arxiv.org/abs/2308.11792 · HTML: https://mlsanity.com/paper/2308.11792/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Practical Transfer Learning for Bayesian Optimization](https://mlsanity.com/paper/1802.02219/) (2018)
- [TrimTuner: Efficient Optimization of Machine Learning Jobs in the Cloud via Sub-Sampling](https://mlsanity.com/paper/2011.04726/) (2020)
- [Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM](https://mlsanity.com/paper/1712.10081/) (2017)
- [Micky: A Cheaper Alternative for Selecting Cloud Instances](https://mlsanity.com/paper/1803.05587/) (2018)
- [C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds](https://mlsanity.com/paper/2107.13317/) (2021)
- [Lynceus: Cost-efficient Tuning and Provisioning of Data Analytic Jobs](https://mlsanity.com/paper/1905.02119/) (2019)
- [Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts](https://mlsanity.com/paper/2107.13921/) (2021)
- [Blink: Lightweight Sample Runs for Cost Optimization of Big Data Applications](https://mlsanity.com/paper/2207.02290/) (2022)
- [Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing](https://mlsanity.com/paper/2211.04240/) (2022)
- [Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics](https://mlsanity.com/paper/2211.08227/) (2022)

## Abstract (arXiv)

Selecting the right resources for big data analytics jobs is hard because of the wide variety of configuration options like machine type and cluster size. As poor choices can have a significant impact on resource efficiency, cost, and energy usage, automated approaches are gaining popularity. Most existing methods rely on profiling recurring workloads to find near-optimal solutions over time. Due to the cold-start problem, this often leads to lengthy and costly profiling phases. However, big data analytics jobs across users can share many common properties: they often operate on similar infrastructure, using similar algorithms implemented in similar frameworks. The potential in sharing aggregated profiling runs to collaboratively address the cold start problem is largely unexplored. We present Karasu, an approach to more efficient resource configuration profiling that promotes data sharing among users working with similar infrastructures, frameworks, algorithms, or datasets. Karasu trains lightweight performance models using aggregated runtime information of collaborators and combines them into an ensemble method to exploit inherent knowledge of the configuration search space. Moreover, Karasu allows the optimization of multiple objectives simultaneously. Our evaluation is based on performance data from diverse workload executions in a public cloud environment. We show that Karasu is able to significantly boost existing methods in terms of performance, search time, and cost, even when few comparable profiling runs are available that share only partial common characteristics with the target job.