mlsanityData to 7 Oct 2026

Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models

arXiv 2104.05158 ↗First posted 12 Apr 2021cs.DC cs.AI cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of Zion platform, namely ZionEX. We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40X speedup in terms of time to solution over previous systems. We achieve this by (i) designing the ZionEX platform with dedicated scale-out network, provisioned with high bandwidth, optimal topology and efficient transport (ii) implementing an optimized PyTorch-based training stack supporting both model and data parallelism (iii) developing sharding algorithms capable of hierarchical partitioning of the embedding tables along row, column dimensions and load balancing them across multiple workers; (iv) adding high-performance core operators while retaining flexibility to support optimizers with fully deterministic updates (v) leveraging reduced precision communications, multi-level memory hierarchy (HBM+DDR+SSD) and pipelining. Furthermore, we develop and briefly comment on distributed data ingestion and other supporting services that are required for the robust and efficient end-to-end training in production environments.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 37 of its 75 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

37 of 75 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2017Neural Collaborative Filteringcited by 1,108
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
25 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  2. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  3. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  4. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  5. 2019MLPerf Inference Benchmarkcited by 195
  6. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  7. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  8. 2019MLPerf Training Benchmarkcited by 114
  9. 2014Deep learning with Elastic Averaging SGDcited by 90
  10. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  11. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
  12. 2017Device Placement Optimization with Reinforcement Learningcited by 84
  13. 2019The Architectural Implications of Facebook's DNN-based Personalized Recommendationcited by 83
  14. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 62
  15. 2020Dynamic Tensor Rematerializationcited by 45
  16. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  17. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 37
  18. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 35
  19. 2020Synthesizing Optimal Collective Algorithmscited by 30
  20. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  21. 2020Mixed-Precision Embedding Using a Cachecited by 18
  22. 2020Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architecturescited by 17
  23. 2020Learning from eXtreme Bandit Feedbackcited by 17
  24. 20183LC: Lightweight and Effective Traffic Compression for Distributed Machine Learningcited by 16
  25. 2020ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Trainingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 16; 2023: 18; 2024: 18; 2025: 13; 2026: 8.

78 papers.

Sort:

By area:

  1. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  2. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  3. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  4. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  5. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  6. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  7. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  8. 2022Understanding Scaling Laws for Recommendation Modelscited by 38
  9. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  10. 2021Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Trainingcited by 28
  11. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  12. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 15
38 more
  1. 2023Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systemscited by 14
  2. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  3. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  4. 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameterscited by 13
  5. 2022DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Predictioncited by 12
  6. 2023Experimental Analysis of Large-scale Learnable Vector Storage Compressioncited by 12
  7. 2022Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwardscited by 11
  8. 2022BagPipe: Accelerating Deep Recommendation Model Trainingcited by 10
  9. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
  10. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  11. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  12. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8
  13. 2025DV365: Extremely Long User History Modeling at Instagramcited by 8
  14. 2023CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Modelscited by 7
  15. 2022Impact of RoCE Congestion Control Policies on Distributed Training of DNNscited by 6
  16. 2024Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendationcited by 6
  17. 2023GPU-based Private Information Retrieval for On-Device Machine Learning Inferencecited by 5
  18. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  19. 2023Sparsity-Preserving Differentially Private Training of Large Embedding Modelscited by 5
  20. 2023Modyn: Data-Centric Machine Learning Pipeline Orchestrationcited by 5
  21. 2024cedar: Optimized and Unified Machine Learning Input Data Pipelinescited by 5
  22. 2025Request-Only Optimization for Recommendation Systemscited by 5
  23. 2022CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPUcited by 4
  24. 2022A Frequency-aware Software Cache for Large Recommendation System Embeddingscited by 4
  25. 2023MCR-DL: Mix-and-Match Communication Runtime for Deep Learningcited by 4
  26. 2023AutoML for Large Capacity Modeling of Meta's Ranking Systemscited by 4
  27. 2024G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systemscited by 4
  28. 2024A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systemscited by 4
  29. 2021LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Modelscited by 3
  30. 2023InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Modelscited by 3
  31. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  32. 2025CLM: Removing the GPU Memory Barrier for 3D Gaussian Splattingcited by 3
  33. 2022COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Trainingcited by 2
  34. 2024PreSto: An In-Storage Data Preprocessing System for Training Recommendation Modelscited by 2
  35. 2024On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspectivecited by 2
  36. 2025Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Trainingcited by 2
  37. 2026ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systemscited by 2
  38. 2022RAMP: A Flat Nanosecond Optical Network and MPI Operations for Distributed Deep Learning Systemscited by 1

50 most cited of 78 All 78 (JSON)