mlsanityData to 14 Oct 2026

Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings

arXiv 2404.04270 ↗First posted 22 Mar 2024cs.IR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Training recommendation models pose significant challenges regarding resource utilization and performance. Prior research has proposed an approach that categorizes embeddings into popular and non-popular classes to reduce the training time for recommendation models. We observe that, even among the popular embeddings, certain embeddings undergo rapid training and exhibit minimal subsequent variation, resulting in saturation. Consequently, updates to these embeddings lack any contribution to model quality. This paper presents Slipstream, a software framework that identifies stale embeddings on the fly and skips their updates to enhance performance. This capability enables Slipstream to achieve substantial speedup, optimize CPU-GPU bandwidth usage, and eliminate unnecessary memory access. SlipStream showcases training time reductions of 2x, 2.4x, 1.2x, and 1.175x across real-world datasets and configurations, compared to Baseline XDL, Intel-optimized DRLM, FAE, and Hotline, respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

22 of 45 references.

References · 22

  1. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  2. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  3. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  4. 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
  5. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  6. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 327
  7. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 320
  8. 2018xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systemscited by 278
  9. 2016Product-based Neural Networks for User Response Predictioncited by 145
  10. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 113
  11. 2019Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systemscited by 68
  12. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
10 more
  1. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  2. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
  3. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  4. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 41
  5. 2017Early Stopping without a Validation Setcited by 39
  6. 2021Early-stopped neural networks are consistentcited by 34
  7. 2020Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architecturescited by 17
  8. 2020Time-based Sequence Model for Personalization and Recommendation Systemscited by 11
  9. 2020A Generic Network Compression Framework for Sequential Recommender Systemscited by 10
  10. 2019Saec: Similarity-Aware Embedding Compression in Recommendation Systemscited by 7

Built on by

Not cited yet.