# Improving Spark Application Throughput Via Memory Aware Task Co-location: A Mixture of Experts Approach

arXiv: https://arxiv.org/abs/1710.00610 · HTML: https://mlsanity.com/paper/1710.00610/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2017.

## Built on by (most cited first, 5 of 5)

- [Supersonic: Learning to Generate Source Code Optimizations in C/C++](https://mlsanity.com/paper/2309.14846/) (2023)
- [On-edge Multi-task Transfer Learning: Model and Practice with Data-driven Task Allocation](https://mlsanity.com/paper/2107.02466/) (2021)
- [PROMPT: Learning Dynamic Resource Allocation Policies for Network Applications](https://mlsanity.com/paper/2201.07916/) (2022)
- [RAPID: Enabling Fast Online Policy Learning in Dynamic Public Cloud Environments](https://mlsanity.com/paper/2304.04797/) (2023)
- [Enhancing Deployment-Time Predictive Model Robustness for Code Analysis and Optimization](https://mlsanity.com/paper/2501.00298/) (2024)

## Abstract (arXiv)

Data analytic applications built upon big data processing frameworks such as Apache Spark are an important class of applications. Many of these applications are not latency-sensitive and thus can run as batch jobs in data centers. By running multiple applications on a computing host, task co-location can significantly improve the server utilization and system throughput. However, effective task co-location is a non-trivial task, as it requires an understanding of the computing resource requirement of the co-running applications, in order to determine what tasks, and how many of them, can be co-located. In this paper, we present a mixture-of-experts approach to model the memory behavior of Spark applications. We achieve this by learning, off-line, a range of specialized memory models on a range of typical applications; we then determine at runtime which of the memory models, or experts, best describes the memory behavior of the target application. We show that by accurately estimating the resource level that is needed, a co-location scheme can effectively determine how many applications can be co-located on the same host to improve the system throughput, by taking into consideration the memory and CPU requirements of co-running application tasks. Our technique is applied to a set of representative data analytic applications built upon the Apache Spark framework. We evaluated our approach for system throughput and average normalized turnaround time on a multi-core cluster. Our approach achieves over 83.9% of the performance delivered using an ideal memory predictor. We obtain, on average, 8.69x improvement on system throughput and a 49% reduction on turnaround time over executing application tasks in isolation, which translates to a 1.28x and 1.68x improvement over a state-of-the-art co-location scheme for system throughput and turnaround time respectively.