# Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism

arXiv: https://arxiv.org/abs/2010.08899 · HTML: https://mlsanity.com/paper/2010.08899/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2020.

## Built on by (most cited first, 15 of 15)

- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Training Large-Scale News Recommenders with Pretrained Language Models in the Loop](https://mlsanity.com/paper/2102.09268/) (2021)
- [Towards Efficient Communication and Secure Federated Recommendation System via Low-rank Training](https://mlsanity.com/paper/2401.03748/) (2024)
- [BagPipe: Accelerating Deep Recommendation Model Training](https://mlsanity.com/paper/2202.12429/) (2022)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance](https://mlsanity.com/paper/2105.08820/) (2021)
- [Activations and Gradients Compression for Model-Parallel Training](https://mlsanity.com/paper/2401.07788/) (2024)
- [GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs](https://mlsanity.com/paper/2407.15452/) (2024)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)
- [ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems](https://mlsanity.com/paper/2608.18469/) (2026)
- [A Machine Learning Framework for Distributed Functional Compression over Wireless Channels in IoT](https://mlsanity.com/paper/2201.09483/) (2022)
- [SCRec: A Scalable Computational Storage System with Statistical Sharding and Tensor-train Decomposition for Recommendation Models](https://mlsanity.com/paper/2504.00520/) (2025)

## Abstract (arXiv)

In this paper, we consider hybrid parallelism -- a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP) -- to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least 100× and 20× during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.