# Wukong: A Scalable and Locality-Enhanced Framework for Serverless Parallel Computing

arXiv: https://arxiv.org/abs/2010.07268 · HTML: https://mlsanity.com/paper/2010.07268/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [How Does It Function? Characterizing Long-term Trends in Production Serverless Workloads](https://mlsanity.com/paper/2312.10127/) (2023)
- [M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining](https://mlsanity.com/paper/2401.15896/) (2024)
- [FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication](https://mlsanity.com/paper/2403.15195/) (2024)
- [Block size estimation for data partitioning in HPC applications using machine learning techniques](https://mlsanity.com/paper/2211.10819/) (2022)
- [High-Performance Serverless Computing: A Systematic Literature Review on Serverless for HPC, AI, and Big Data](https://mlsanity.com/paper/2601.09334/) (2026)

## Abstract (arXiv)

Serverless computing is increasingly being used for parallel computing, which have traditionally been implemented as stateful applications. Executing complex, burst-parallel, directed acyclic graph (DAG) jobs poses a major challenge for serverless execution frameworks, which will need to rapidly scale and schedule tasks at high throughput, while minimizing data movement across tasks. We demonstrate that, for serverless parallel computations, decentralized scheduling enables scheduling to be distributed across Lambda executors that can schedule tasks in parallel, and brings multiple benefits, including enhanced data locality, reduced network I/Os, automatic resource elasticity, and improved cost effectiveness. We describe the implementation and deployment of our new serverless parallel framework, called Wukong, on AWS Lambda. We show that Wukong achieves near-ideal scalability, executes parallel computation jobs up to 68.17x faster, reduces network I/O by multiple orders of magnitude, and achieves 92.96% tenant-side cost savings compared to numpywren.