# On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

arXiv: https://arxiv.org/abs/2111.08759 · HTML: https://mlsanity.com/paper/2111.08759/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 16 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Micky: A Cheaper Alternative for Selecting Cloud Instances](https://mlsanity.com/paper/1803.05587/) (2018)
- [Characterizing Co-located Datacenter Workloads: An Alibaba Case Study](https://mlsanity.com/paper/1808.02919/) (2018)
- [C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds](https://mlsanity.com/paper/2107.13317/) (2021)
- [Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts](https://mlsanity.com/paper/2107.13921/) (2021)
- [Parallelization in Scientific Workflow Management Systems](https://mlsanity.com/paper/1303.7195/) (2013)
- [Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs](https://mlsanity.com/paper/2011.07965/) (2020)
- [Tarema: Adaptive Resource Allocation for Scalable Scientific Workflows in Heterogeneous Clusters](https://mlsanity.com/paper/2111.05167/) (2021)
- [Training Data Reduction for Performance Models of Data Analytics Jobs in the Cloud](https://mlsanity.com/paper/2111.07904/) (2021)

## Built on by (most cited first, 1 of 1)

- [ReAssigner: A Plug-and-Play Virtual Machine Scheduling Intensifier for Heterogeneous Requests](https://mlsanity.com/paper/2211.16227/) (2022)

## Abstract (arXiv)

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected. In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.