# C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds

arXiv: https://arxiv.org/abs/2107.13317 · HTML: https://mlsanity.com/paper/2107.13317/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2021.

## Built on by (most cited first, 5 of 5)

- [Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts](https://mlsanity.com/paper/2107.13921/) (2021)
- [Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation](https://mlsanity.com/paper/2108.12211/) (2021)
- [On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds](https://mlsanity.com/paper/2111.08759/) (2021)
- [Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics](https://mlsanity.com/paper/2211.08227/) (2022)
- [Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics](https://mlsanity.com/paper/2308.11792/) (2023)

## Abstract (arXiv)

Distributed dataflow systems enable data-parallel processing of large datasets on clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. Yet, selecting appropriate cloud resources for dataflow jobs - that neither lead to bottlenecks nor to low resource utilization - is often challenging, even for expert users such as data engineers. We present C3O, a collaborative system for optimizing data processing cluster configurations in public clouds based on shared historical runtime data. The shared data is utilized for predicting the runtimes of data processing jobs on different possible cluster configurations, using specialized regression models. These models take the diverse execution contexts of different users into account and exhibit mean absolute errors below 3% in our experimental evaluation with 930 unique Spark jobs.