# Understanding Capacity-Driven Scale-Out Neural Recommendation Inference

arXiv: https://arxiv.org/abs/2011.02084 · HTML: https://mlsanity.com/paper/2011.02084/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 29 of 29)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference](https://mlsanity.com/paper/2102.00075/) (2021)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendation](https://mlsanity.com/paper/2201.10095/) (2022)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards](https://mlsanity.com/paper/2205.04702/) (2022)
- [The Unseen AI Disruptions for Power Grids: LLM-Induced Transients](https://mlsanity.com/paper/2409.11416/) (2024)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance](https://mlsanity.com/paper/2105.08820/) (2021)
- [Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale](https://mlsanity.com/paper/2105.12676/) (2021)
- [Supporting Massive DLRM Inference Through Software Defined Memory](https://mlsanity.com/paper/2110.11489/) (2021)
- [A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities](https://mlsanity.com/paper/2111.14247/) (2021)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learning](https://mlsanity.com/paper/2206.03852/) (2022)
- [FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Models](https://mlsanity.com/paper/2301.02959/) (2023)
- [RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detection](https://mlsanity.com/paper/2503.22733/) (2025)
- [Data Leakage via Access Patterns of Sparse Features in Deep Learning-based Recommendation Systems](https://mlsanity.com/paper/2212.06264/) (2022)
- [Evaluating and Enhancing Robustness of Deep Recommendation Systems Against Hardware Errors](https://mlsanity.com/paper/2307.10244/) (2023)
- [Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms](https://mlsanity.com/paper/2404.12674/) (2024)
- [CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data](https://mlsanity.com/paper/2407.08108/) (2024)
- [Disaggregating Embedding Recommendation Systems with FlexEMR](https://mlsanity.com/paper/2410.12794/) (2024)
- [Rec-AD: An Efficient Computation Framework for FDIA Detection Based on Tensor Train Decomposition and Deep Learning Recommendation Model](https://mlsanity.com/paper/2507.14668/) (2025)
- [ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding](https://mlsanity.com/paper/2104.01981/) (2021)
- [Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations](https://mlsanity.com/paper/2302.11750/) (2023)
- [ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models](https://mlsanity.com/paper/2406.06955/) (2024)
- [Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs](https://mlsanity.com/paper/2410.22249/) (2024)

## Abstract (arXiv)

Deep learning recommendation models have grown to the terabyte scale. Traditional serving schemes--that load entire models to a single server--are unable to support this scale. One approach to support this scale is with distributed serving, or distributed inference, which divides the memory requirements of a single large model across multiple servers. This work is a first-step for the systems research community to develop novel model-serving solutions, given the huge system design space. Large-scale deep recommender systems are a novel workload and vital to study, as they consume up to 79% of all inference cycles in the data center. To that end, this work describes and characterizes scale-out deep learning recommendation inference using data-center serving infrastructure. This work specifically explores latency-bounded inference systems, compared to the throughput-oriented training systems of other recent works. We find that the latency and compute overheads of distributed inference are largely a result of a model's static embedding table distribution and sparsity of input inference requests. We further evaluate three embedding table mapping strategies of three DLRM-like models and specify challenging design trade-offs in terms of end-to-end latency, compute overhead, and resource efficiency. Overall, we observe only a marginal latency overhead when the data-center scale recommendation models are served with the distributed inference manner--P99 latency is increased by only 1% in the best case configuration. The latency overheads are largely a result of the commodity infrastructure used and the sparsity of embedding tables. Even more encouragingly, we also show how distributed inference can account for efficiency improvements in data-center scale recommendation serving.