# A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models

arXiv: https://arxiv.org/abs/2210.08804 · HTML: https://mlsanity.com/paper/2210.08804/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference](https://mlsanity.com/paper/2001.02772/) (2020)
- [RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference](https://mlsanity.com/paper/2102.00075/) (2021)
- [Understanding Capacity-Driven Scale-Out Neural Recommendation Inference](https://mlsanity.com/paper/2011.02084/) (2020)
- [Distributed Training of Deep Learning Models: A Taxonomic Perspective](https://mlsanity.com/paper/2007.03970/) (2020)
- [Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training](https://mlsanity.com/paper/1805.07891/) (2018)
- [Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism](https://mlsanity.com/paper/2010.08899/) (2020)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)

## Built on by (most cited first, 9 of 9)

- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training](https://mlsanity.com/paper/2504.09307/) (2025)
- [InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models](https://mlsanity.com/paper/2308.08500/) (2023)
- [Toward Robust and Efficient ML-Based GPU Caching for Modern Inference](https://mlsanity.com/paper/2509.20979/) (2025)
- [xGR: Efficient Generative Recommendation Serving at Scale](https://mlsanity.com/paper/2512.11529/) (2025)
- [RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference](https://mlsanity.com/paper/2601.01712/) (2026)
- [Disaggregating Embedding Recommendation Systems with FlexEMR](https://mlsanity.com/paper/2410.12794/) (2024)
- [Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates](https://mlsanity.com/paper/2512.12295/) (2025)
- [Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended](https://mlsanity.com/paper/2605.30728/) (2026)

## Abstract (arXiv)

Recommendation systems are of crucial importance for a variety of modern apps and web services, such as news feeds, social networks, e-commerce, search, etc. To achieve peak prediction accuracy, modern recommendation models combine deep learning with terabyte-scale embedding tables to obtain a fine-grained representation of the underlying data. Traditional inference serving architectures require deploying the whole model to standalone servers, which is infeasible at such massive scale. In this paper, we provide insights into the intriguing and challenging inference domain of online recommendation systems. We propose the HugeCTR Hierarchical Parameter Server (HPS), an industry-leading distributed recommendation inference framework, that combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. Among other things, HPS features (1) a redundant hierarchical storage system, (2) a novel high-bandwidth cache to accelerate parallel embedding lookup on NVIDIA GPUs, (3) online training support and (4) light-weight APIs for easy integration into existing large-scale recommendation workflows. To demonstrate its capabilities, we conduct extensive studies using both synthetically engineered and public datasets. We show that our HPS can dramatically reduce end-to-end inference latency, achieving 5~62x speedup (depending on the batch size) over CPU baseline implementations for popular recommendation models. Through multi-GPU concurrent deployment, the HPS can also greatly increase the inference QPS.