# NuPS: A Parameter Server for Machine Learning with Non-Uniform Parameter Access

arXiv: https://arxiv.org/abs/2104.00501 · HTML: https://mlsanity.com/paper/2104.00501/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 5 of 5)

- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Understanding Negative Sampling in Graph Representation Learning](https://mlsanity.com/paper/2005.09863/) (2020)
- [Petuum: A New Platform for Distributed Machine Learning on Big Data](https://mlsanity.com/paper/1312.7651/) (2013)
- [Dynamic Parameter Allocation in Parameter Servers](https://mlsanity.com/paper/2002.00655/) (2020)
- [Computing Web-scale Topic Models using an Asynchronous Parameter Server](https://mlsanity.com/paper/1605.07422/) (2016)

## Built on by (most cited first, 7 of 7)

- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [The Duck's Brain: Training and Inference of Neural Networks in Modern Database Engines](https://mlsanity.com/paper/2312.17355/) (2023)
- [Good Intentions: Adaptive Parameter Management via Intent Signaling](https://mlsanity.com/paper/2206.00470/) (2022)
- [SparDL: Distributed Deep Learning Training with Efficient Sparse Communication](https://mlsanity.com/paper/2304.00737/) (2023)
- [MLKV: Efficiently Scaling up Large Embedding Model Training with Disk-based Key-Value Storage](https://mlsanity.com/paper/2504.01506/) (2025)
- [NOMAD: Generating Embeddings for Massive Distributed Graphs](https://mlsanity.com/paper/2604.09419/) (2026)
- [FeLoG: Scalable and Efficient Distributed Graph Embedding with Feedback Loop Mechanism](https://mlsanity.com/paper/2606.22180/) (2026)

## Abstract (arXiv)

Parameter servers (PSs) facilitate the implementation of distributed training for large machine learning tasks. In this paper, we argue that existing PSs are inefficient for tasks that exhibit non-uniform parameter access; their performance may even fall behind that of single node baselines. We identify two major sources of such non-uniform access: skew and sampling. Existing PSs are ill-suited for managing skew because they uniformly apply the same parameter management technique to all parameters. They are inefficient for sampling because the PS is oblivious to the associated randomized accesses and cannot exploit locality. To overcome these performance limitations, we introduce NuPS, a novel PS architecture that (i) integrates multiple management techniques and employs a suitable technique for each parameter and (ii) supports sampling directly via suitable sampling primitives and sampling schemes that allow for a controlled quality--efficiency trade-off. In our experimental study, NuPS outperformed existing PSs by up to one order of magnitude and provided up to linear scalability across multiple machine learning tasks.