mlsanityData to 14 Oct 2026

Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems

arXiv 2003.09518 ↗First posted 20 Mar 2020cs.DCcited by 38

Not on the map. Why?

Abstract (arXiv)

Large-scale training is important to ensure high performance and accuracy of machine-learning models. At Facebook we use many different models, including computer vision, video and language models. However, in this paper we focus on the deep learning recommendation models (DLRMs), which are responsible for more than 50% of the training demand in our data centers. Recommendation models present unique challenges in training because they exercise not only compute but also memory capacity as well as memory and network bandwidth. As model size and complexity increase, efficiently scaling training becomes a challenge. To address it we design Zion - Facebook's next-generation large-memory training platform that consists of both CPUs and accelerators. Also, we discuss the design requirements of future scale-out training systems.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 13; 2023: 4; 2024: 5; 2025: 6; 2026: 1.

38 papers.

Sort:

By area:

  1. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  2. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  3. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  4. 2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
  5. 2022Understanding Scaling Laws for Recommendation Modelscited by 38
  6. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  7. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28
  8. 2021Accelerating Recommendation System Training by Leveraging Popular Choicescited by 22
  9. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  10. 2022AutoShard: Automated Embedding Table Sharding for Recommender Systemscited by 17
  11. 2022DreamShard: Generalizable Embedding Table Placement for Recommender Systemscited by 14
  12. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
26 more
  1. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  2. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
  3. 2021RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performancecited by 8
  4. 2024Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendationcited by 6
  5. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  6. 2021Supporting Massive DLRM Inference Through Software Defined Memorycited by 5
  7. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
  8. 2022Learning to Collide: Recommendation System Model Compression with Learned Hash Functionscited by 5
  9. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  10. 2022DNNAbacus: Toward Accurate Computational Cost Prediction for Deep Neural Networkscited by 4
  11. 2023FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Modelscited by 4
  12. 2024UpDLRM: Accelerating Personalized Recommendation using Real-World PIM Architecturecited by 4
  13. 2021Alternate Model Growth and Pruning for Efficient Training of Recommendation Systemscited by 3
  14. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  15. 2022The trade-offs of model size in large recommendation models : A 10000 × compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)cited by 2
  16. 2022COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Trainingcited by 2
  17. 2025Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Trainingcited by 2
  18. 2022RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instancescited by 1
  19. 2024CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Datacited by 1
  20. 2025Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updatescited by 1
  21. 2023MTrainS: Improving DLRM training efficiency using heterogeneous memoriescited by 0
  22. 2023On the Burstiness of Distributed Machine Learning Trafficcited by 0
  23. 2025Routing for Large ML Modelscited by 0
  24. 2025SCRec: A Scalable Computational Storage System with Statistical Sharding and Tensor-train Decomposition for Recommendation Modelscited by 0
  25. 2025Make It Long, Keep It Fast: End-to-End 10K Long User Behavior Sequence Modeling for Billion-Scale Douyin Recommendationcited by 0
  26. 2026Deploying RDMA for Distributed Inference and Training At Scalecited by 0