# Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training

arXiv: https://arxiv.org/abs/1805.07891 · HTML: https://mlsanity.com/paper/1805.07891/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 17 of 17)

- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://mlsanity.com/paper/2411.04996/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Quantum Federated Learning: A Comprehensive Survey](https://mlsanity.com/paper/2508.15998/) (2025)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [Shifted Compression Framework: Generalizations and Improvements](https://mlsanity.com/paper/2206.10452/) (2022)
- [Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation](https://mlsanity.com/paper/2403.00877/) (2024)
- [Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising](https://mlsanity.com/paper/2201.05500/) (2022)
- [Accelerating Distributed Deep Learning using Lossless Homomorphic Compression](https://mlsanity.com/paper/2402.07529/) (2024)
- [MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training](https://mlsanity.com/paper/2103.15195/) (2021)
- [ByteComp: Revisiting Gradient Compression in Distributed Training](https://mlsanity.com/paper/2205.14465/) (2022)
- [Fair and Efficient Distributed Edge Learning with Hybrid Multipath TCP](https://mlsanity.com/paper/2211.09723/) (2022)
- [LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models](https://mlsanity.com/paper/2404.08847/) (2024)
- [LoKA: Low-precision Kernel Applications for Recommendation Models At Scale](https://mlsanity.com/paper/2605.10886/) (2026)
- [Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering](https://mlsanity.com/paper/2105.14088/) (2021)
- [FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost](https://mlsanity.com/paper/2604.24073/) (2026)

## Abstract (arXiv)

Distributed deep neural network (DDNN) training constitutes an increasingly important workload that frequently runs in the cloud. Larger DNN models and faster compute engines are shifting DDNN training bottlenecks from computation to communication. This paper characterizes DDNN training to precisely pinpoint these bottlenecks. We found that timely training requires high performance parameter servers (PSs) with optimized network stacks and gradient processing pipelines, as well as server and network hardware with balanced computation and communication resources. We therefore propose PHub, a high performance multi-tenant, rack-scale PS design. PHub co-designs the PS software and hardware to accelerate rack-level and hierarchical cross-rack parameter exchange, with an API compatible with many DDNN training frameworks. PHub provides a performance improvement of up to 2.7x compared to state-of-the-art distributed training techniques for cloud-based ImageNet workloads, with 25% better throughput per dollar.