# Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings

arXiv: https://arxiv.org/abs/2404.04270 · HTML: https://mlsanity.com/paper/2404.04270/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Understanding and Improving Early Stopping for Learning with Noisy Labels](https://mlsanity.com/paper/2106.15853/) (2021)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing](https://mlsanity.com/paper/1912.12953/) (2019)
- [Understanding Training Efficiency of Deep Learning Recommendation Models at Scale](https://mlsanity.com/paper/2011.05497/) (2020)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [Early Stopping without a Validation Set](https://mlsanity.com/paper/1703.09580/) (2017)
- [Early-stopped neural networks are consistent](https://mlsanity.com/paper/2106.05932/) (2021)
- [Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures](https://mlsanity.com/paper/2005.04680/) (2020)
- [Time-based Sequence Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/2008.11922/) (2020)
- [A Generic Network Compression Framework for Sequential Recommender Systems](https://mlsanity.com/paper/2004.13139/) (2020)
- [Saec: Similarity-Aware Embedding Compression in Recommendation Systems](https://mlsanity.com/paper/1903.00103/) (2019)

## Abstract (arXiv)

Training recommendation models pose significant challenges regarding resource utilization and performance. Prior research has proposed an approach that categorizes embeddings into popular and non-popular classes to reduce the training time for recommendation models. We observe that, even among the popular embeddings, certain embeddings undergo rapid training and exhibit minimal subsequent variation, resulting in saturation. Consequently, updates to these embeddings lack any contribution to model quality. This paper presents Slipstream, a software framework that identifies stale embeddings on the fly and skips their updates to enhance performance. This capability enables Slipstream to achieve substantial speedup, optimize CPU-GPU bandwidth usage, and eliminate unnecessary memory access. SlipStream showcases training time reductions of 2x, 2.4x, 1.2x, and 1.175x across real-world datasets and configurations, compared to Baseline XDL, Intel-optimized DRLM, FAE, and Hotline, respectively.