mlsanityData to 7 Oct 2026

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

arXiv 2011.05497 ↗First posted 11 Nov 2020cs.AR cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale datacenters, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components. GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 13; 2023: 10; 2024: 5; 2025: 2; 2026: 1.

43 papers.

Sort:

  1. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  2. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  3. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  4. 2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
  5. 2022Understanding Scaling Laws for Recommendation Modelscited by 38
  6. 2021Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Trainingcited by 28
  7. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28
  8. 2022A Survey on Federated Recommendation Systemscited by 25
  9. 2021Accelerating Recommendation System Training by Leveraging Popular Choicescited by 22
  10. 2021Modeling Techniques for Machine Learning Fairness: A Surveycited by 17
  11. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  12. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
31 more
  1. 2022AutoShard: Automated Embedding Table Sharding for Recommender Systemscited by 17
  2. 2022DreamShard: Generalizable Embedding Table Placement for Recommender Systemscited by 14
  3. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  4. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  5. 2022BagPipe: Accelerating Deep Recommendation Model Trainingcited by 10
  6. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
  7. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  8. 2021RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performancecited by 8
  9. 2023Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender Systemcited by 7
  10. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  11. 2022GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Modelcited by 4
  12. 2022FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learningcited by 4
  13. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  14. 2023FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Modelscited by 4
  15. 2021Alternate Model Growth and Pruning for Efficient Training of Recommendation Systemscited by 3
  16. 2023Mem-Rec: Memory Efficient Recommendation System using Alternative Representationcited by 3
  17. 2023InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Modelscited by 3
  18. 2022COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Trainingcited by 2
  19. 2023Saturn: An Optimized Data System for Large Model Deep Learning Workloadscited by 2
  20. 2024PreSto: An In-Storage Data Preprocessing System for Training Recommendation Modelscited by 2
  21. 2024Data Deletion for Linear Regression with Noisy SGDcited by 2
  22. 2021Boost-RS: Boosted Embeddings for Recommender Systems and its Application to Enzyme-Substrate Interaction Predictioncited by 1
  23. 2022Data Leakage via Access Patterns of Sparse Features in Deep Learning-based Recommendation Systemscited by 1
  24. 2023Evaluating and Enhancing Robustness of Deep Recommendation Systems Against Hardware Errorscited by 1
  25. 2024CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Datacited by 1
  26. 2025Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updatescited by 1
  27. 2021ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Codingcited by 0
  28. 2024Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddingscited by 0
  29. 2024ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Modelscited by 0
  30. 2025Ember: A Compiler for Efficient Embedding Operations on Decoupled Access-Execute Architecturescited by 0
  31. 2026Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extendedcited by 0