# DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference

arXiv: https://arxiv.org/abs/2001.02772 · HTML: https://mlsanity.com/paper/2001.02772/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2020.

## Built on by (most cited first, 30 of 54)

- [S3: Increasing GPU Utilization during Generative Inference for Higher Throughput](https://mlsanity.com/paper/2306.06000/) (2023)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services](https://mlsanity.com/paper/2404.16283/) (2024)
- [RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference](https://mlsanity.com/paper/2102.00075/) (2021)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendation](https://mlsanity.com/paper/2201.10095/) (2022)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores](https://mlsanity.com/paper/2104.13997/) (2021)
- [ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach](https://mlsanity.com/paper/2311.16136/) (2023)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Heterogeneous Acceleration Pipeline for Recommendation System Training](https://mlsanity.com/paper/2204.05436/) (2022)
- [Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards](https://mlsanity.com/paper/2205.04702/) (2022)
- [BagPipe: Accelerating Deep Recommendation Model Training](https://mlsanity.com/paper/2202.12429/) (2022)
- [On Estimating Recommendation Evaluation Metrics under Sampling](https://mlsanity.com/paper/2103.01474/) (2021)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance](https://mlsanity.com/paper/2105.08820/) (2021)
- [Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale](https://mlsanity.com/paper/2105.12676/) (2021)
- [CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models](https://mlsanity.com/paper/2312.03256/) (2023)
- [Differentiable NAS Framework and Application to Ads CTR Prediction](https://mlsanity.com/paper/2110.14812/) (2021)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Learning to Collide: Recommendation System Model Compression with Learned Hash Functions](https://mlsanity.com/paper/2203.15837/) (2022)
- [GPU-based Private Information Retrieval for On-Device Machine Learning Inference](https://mlsanity.com/paper/2301.10904/) (2023)
- [Request-Only Optimization for Recommendation Systems](https://mlsanity.com/paper/2508.05640/) (2025)
- [FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Models](https://mlsanity.com/paper/2301.02959/) (2023)
- [Demystifying BERT: Implications for Accelerator Design](https://mlsanity.com/paper/2104.08335/) (2021)
- [Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems](https://mlsanity.com/paper/2105.01064/) (2021)
- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)
- [Mem-Rec: Memory Efficient Recommendation System using Alternative Representation](https://mlsanity.com/paper/2305.07205/) (2023)

## Abstract (arXiv)

Neural personalized recommendation is the corner-stone of a wide collection of cloud services and products, constituting significant compute demand of the cloud infrastructure. Thus, improving the execution efficiency of neural recommendation directly translates into infrastructure capacity saving. In this paper, we devise a novel end-to-end modeling infrastructure, DeepRecInfra, that adopts an algorithm and system co-design methodology to custom-design systems for recommendation use cases. Leveraging the insights from the recommendation characterization, a new dynamic scheduler, DeepRecSched, is proposed to maximize latency-bounded throughput by taking into account characteristics of inference query size and arrival patterns, recommendation model architectures, and underlying hardware systems. By doing so, system throughput is doubled across the eight industry-representative recommendation models. Finally, design, deployment, and evaluation in at-scale production datacenter shows over 30% latency reduction across a wide variety of recommendation models running on hundreds of machines.