# Stochastic Communication Avoidance for Recommendation Systems

arXiv: https://arxiv.org/abs/2411.01611 · HTML: https://mlsanity.com/paper/2411.01611/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 22 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [DAPPLE: A Pipelined Data Parallel Approach for Training Large Models](https://mlsanity.com/paper/2007.01045/) (2020)
- [Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization](https://mlsanity.com/paper/1910.02653/) (2019)
- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers](https://mlsanity.com/paper/2211.16056/) (2022)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [Post-Training 4-bit Quantization on Embedding Tables](https://mlsanity.com/paper/1911.02079/) (2019)
- [Embedding Compression in Recommender Systems: A Survey](https://mlsanity.com/paper/2408.02304/) (2024)
- [Time-based Sequence Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/2008.11922/) (2020)
- [BagPipe: Accelerating Deep Recommendation Model Training](https://mlsanity.com/paper/2202.12429/) (2022)
- [QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources](https://mlsanity.com/paper/2310.07147/) (2023)
- [Analysis of Quantization on MLP-based Vision Models](https://mlsanity.com/paper/2209.06383/) (2022)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)

## Abstract (arXiv)

One of the major bottlenecks for efficient deployment of neural network based recommendation systems is the memory footprint of their embedding tables. Although many neural network based recommendation systems could benefit from the faster on-chip memory access and increased computational power of hardware accelerators, the large embedding tables in these models often cannot fit on the constrained memory of accelerators. Despite the pervasiveness of these models, prior methods in memory optimization and parallelism fail to address the memory and communication costs of large embedding tables on accelerators. As a result, the majority of models are trained on CPUs, while current implementations of accelerators are hindered by issues such as bottlenecks in inter-device communication and main memory lookups. In this paper, we propose a theoretical framework that analyses the communication costs of arbitrary distributed systems that use lookup tables. We use this framework to propose algorithms that maximize throughput subject to memory, computation, and communication constraints. Furthermore, we demonstrate that our method achieves strong theoretical performance across dataset distributions and memory constraints, applicable to a wide range of use cases from mobile federated learning to warehouse-scale computation. We implement our framework and algorithms in PyTorch and achieve up to 6x increases in training throughput on GPU systems over baselines, on the Criteo Terabytes dataset.