# High-Performance Distributed ML at Scale through Parameter Server Consistency Models

arXiv: https://arxiv.org/abs/1410.8043 · HTML: https://mlsanity.com/paper/1410.8043/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2014.

## Built on by (most cited first, 10 of 10)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [BAGUA: Scaling up Distributed Learning with System Relaxations](https://mlsanity.com/paper/2107.01499/) (2021)
- [FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion](https://mlsanity.com/paper/2311.09540/) (2023)
- [AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning](https://mlsanity.com/paper/2106.14126/) (2021)
- [Order Optimal Bounds for One-Shot Federated Learning over non-Convex Loss Functions](https://mlsanity.com/paper/2108.08677/) (2021)
- [Distributed data analytics](https://mlsanity.com/paper/2203.14088/) (2022)
- [Good Intentions: Adaptive Parameter Management via Intent Signaling](https://mlsanity.com/paper/2206.00470/) (2022)
- [Tackling Computational Heterogeneity in FL: A Few Theoretical Insights](https://mlsanity.com/paper/2307.06283/) (2023)

## Abstract (arXiv)

As Machine Learning (ML) applications increase in data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory demands. Unfortunately, effective use of clusters for ML requires considerable expertise in writing distributed code, while highly-abstracted frameworks like Hadoop have not, in practice, approached the performance seen in specialized ML implementations. The recent Parameter Server (PS) paradigm is a middle ground between these extremes, allowing easy conversion of single-machine parallel ML applications into distributed ones, while maintaining high throughput through relaxed "consistency models" that allow inconsistent parameter reads. However, due to insufficient theoretical study, it is not clear which of these consistency models can really ensure correct ML algorithm output; at the same time, there remain many theoretically-motivated but undiscovered opportunities to maximize computational throughput. Motivated by this challenge, we study both the theoretical guarantees and empirical behavior of iterative-convergent ML algorithms in existing PS consistency models. We then use the gleaned insights to improve a consistency model using an "eager" PS communication mechanism, and implement it as a new PS system that enables ML algorithms to reach their solution more quickly.