# Petuum: A New Platform for Distributed Machine Learning on Big Data

arXiv: https://arxiv.org/abs/1312.7651 · HTML: https://mlsanity.com/paper/1312.7651/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2013.

## Built on by (most cited first, 30 of 35)

- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache](https://mlsanity.com/paper/2401.02669/) (2024)
- [LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models](https://mlsanity.com/paper/2309.14393/) (2023)
- [GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning](https://mlsanity.com/paper/2105.00529/) (2021)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Fast Federated Learning by Balancing Communication Trade-Offs](https://mlsanity.com/paper/2105.11028/) (2021)
- [Distributed Deep Learning Using Volunteer Computing-Like Paradigm](https://mlsanity.com/paper/2103.08894/) (2021)
- [On-edge Multi-task Transfer Learning: Model and Practice with Data-driven Task Allocation](https://mlsanity.com/paper/2107.02466/) (2021)
- [NuPS: A Parameter Server for Machine Learning with Non-Uniform Parameter Access](https://mlsanity.com/paper/2104.00501/) (2021)
- [SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training](https://mlsanity.com/paper/2504.14519/) (2025)
- [ZenFlow: Enabling Stall-Free Offloading for LLM Training](https://mlsanity.com/paper/2505.12242/) (2025)
- [Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment](https://mlsanity.com/paper/2312.03549/) (2023)
- [Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent](https://mlsanity.com/paper/2001.05918/) (2020)
- [Stochastic Gradient Descent without Full Data Shuffle](https://mlsanity.com/paper/2206.05830/) (2022)
- [Baechi: Fast Device Placement of Machine Learning Graphs](https://mlsanity.com/paper/2301.08695/) (2023)
- [Distributed Extra-gradient with Optimal Complexity and Communication Guarantees](https://mlsanity.com/paper/2308.09187/) (2023)
- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism](https://mlsanity.com/paper/2507.00394/) (2025)
- [Sampling Streaming Data with Parallel Vector Quantization -- PVQ](https://mlsanity.com/paper/2210.01792/) (2022)
- [Scaling up Stochastic Gradient Descent for Non-convex Optimisation](https://mlsanity.com/paper/2210.02882/) (2022)
- [AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes](https://mlsanity.com/paper/2404.09679/) (2024)
- [Coded Deep Learning: Framework and Algorithm](https://mlsanity.com/paper/2501.09849/) (2025)
- [Fog Intelligence for Network Anomaly Detection](https://mlsanity.com/paper/2505.21563/) (2025)
- [CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks](https://mlsanity.com/paper/2509.19855/) (2025)
- [Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap](https://mlsanity.com/paper/2512.10236/) (2025)
- [NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization](https://mlsanity.com/paper/2104.13818/) (2021)
- [Good Intentions: Adaptive Parameter Management via Intent Signaling](https://mlsanity.com/paper/2206.00470/) (2022)
- [Quantized Adaptive Subgradient Algorithms and Their Applications](https://mlsanity.com/paper/2208.05631/) (2022)
- [Towards a Flexible and High-Fidelity Approach to Distributed DNN Training Emulation](https://mlsanity.com/paper/2405.02969/) (2024)
- [Hybrid Approach to Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/2407.00101/) (2024)

## Abstract (arXiv)

What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework that systematically addresses data- and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of modern ML algorithms, allowing ML programs to run in much less time and at considerably larger model sizes, even on modestly-sized compute clusters.