# Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale

arXiv: https://arxiv.org/abs/2105.12676 · HTML: https://mlsanity.com/paper/2105.12676/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference](https://mlsanity.com/paper/2005.03842/) (2020)
- [Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks](https://mlsanity.com/paper/1903.08066/) (2019)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [NVIDIA Tensor Core Programmability, Performance & Precision](https://mlsanity.com/paper/1803.04014/) (2018)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model](https://mlsanity.com/paper/1906.00532/) (2019)
- [DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference](https://mlsanity.com/paper/2001.02772/) (2020)
- [Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization](https://mlsanity.com/paper/1902.01917/) (2019)
- [FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference](https://mlsanity.com/paper/2101.05615/) (2021)
- [Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications](https://mlsanity.com/paper/1811.09886/) (2018)
- [Understanding Capacity-Driven Scale-Out Neural Recommendation Inference](https://mlsanity.com/paper/2011.02084/) (2020)
- [Tensor Methods and Recommender Systems](https://mlsanity.com/paper/1603.06038/) (2016)

## Built on by (most cited first, 7 of 7)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [Tango: rethinking quantization for graph neural network training on GPUs](https://mlsanity.com/paper/2308.00890/) (2023)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)
- [Characterizing and Efficiently Accelerating Multimodal Generation Model Inference](https://mlsanity.com/paper/2410.00215/) (2024)
- [Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations](https://mlsanity.com/paper/2302.11750/) (2023)

## Abstract (arXiv)

Tremendous success of machine learning (ML) and the unabated growth in ML model complexity motivated many ML-specific designs in both CPU and accelerator architectures to speed up the model inference. While these architectures are diverse, highly optimized low-precision arithmetic is a component shared by most. Impressive compute throughputs are indeed often exhibited by these architectures on benchmark ML models. Nevertheless, production models such as recommendation systems important to Facebook's personalization services are demanding and complex: These systems must serve billions of users per month responsively with low latency while maintaining high prediction accuracy, notwithstanding computations with many tens of billions parameters per inference. Do these low-precision architectures work well with our production recommendation systems? They do. But not without significant effort. We share in this paper our search strategies to adapt reference recommendation models to low-precision hardware, our optimization of low-precision compute kernels, and the design and development of tool chain so as to maintain our models' accuracy throughout their lifespan during which topic trends and users' interests inevitably evolve. Practicing these low-precision technologies helped us save datacenter capacities while deploying models with up to 5X complexity that would otherwise not be deployed on traditional general-purpose CPUs. We believe these lessons from the trenches promote better co-design between hardware architecture and software engineering and advance the state of the art of ML in industry.