# FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference

arXiv: https://arxiv.org/abs/2101.05615 · HTML: https://mlsanity.com/paper/2101.05615/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 11 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 1 of 1)

- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)

## Built on by (most cited first, 30 of 41)

- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [MagicPIG: LSH Sampling for Efficient LLM Generation](https://mlsanity.com/paper/2410.16179/) (2024)
- [Stable and low-precision training for large-scale vision-language models](https://mlsanity.com/paper/2304.13013/) (2023)
- [DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories](https://mlsanity.com/paper/2111.04625/) (2021)
- [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](https://mlsanity.com/paper/2112.08429/) (2021)
- [AutoShard: Automated Embedding Table Sharding for Recommender Systems](https://mlsanity.com/paper/2208.06399/) (2022)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Revisiting Neural Retrieval on Accelerators](https://mlsanity.com/paper/2306.04039/) (2023)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [FPUS23: An Ultrasound Fetus Phantom Dataset with Deep Neural Network Evaluations for Fetus Orientations, Fetal Planes, and Anatomical Features](https://mlsanity.com/paper/2303.07852/) (2023)
- [PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platform](https://mlsanity.com/paper/2507.12704/) (2025)
- [Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale](https://mlsanity.com/paper/2105.12676/) (2021)
- [A Practical Stereo Depth System for Smart Glasses](https://mlsanity.com/paper/2211.10551/) (2022)
- [Model Pruning Based on Quantified Similarity of Feature Maps](https://mlsanity.com/paper/2105.06052/) (2021)
- [Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation](https://mlsanity.com/paper/2403.00877/) (2024)
- [Supporting Massive DLRM Inference Through Software Defined Memory](https://mlsanity.com/paper/2110.11489/) (2021)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [μnit Scaling: Simple and Scalable FP8 LLM Training](https://mlsanity.com/paper/2502.05967/) (2025)
- [Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models](https://mlsanity.com/paper/2103.00130/) (2021)
- [An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference](https://mlsanity.com/paper/2605.07719/) (2026)
- [The NiuTrans System for the WMT21 Efficiency Task](https://mlsanity.com/paper/2109.08003/) (2021)
- [Verifiable and Energy Efficient Medical Image Analysis with Quantised Self-attentive Deep Neural Networks](https://mlsanity.com/paper/2209.15287/) (2022)
- [Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training](https://mlsanity.com/paper/2508.03854/) (2025)
- [Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments](https://mlsanity.com/paper/2311.05970/) (2023)
- [Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms](https://mlsanity.com/paper/2404.12674/) (2024)
- [Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates](https://mlsanity.com/paper/2512.12295/) (2025)
- [LoKA: Low-precision Kernel Applications for Recommendation Models At Scale](https://mlsanity.com/paper/2605.10886/) (2026)

## Abstract (arXiv)

Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.