# Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures

arXiv: https://arxiv.org/abs/2005.04680 · HTML: https://mlsanity.com/paper/2005.04680/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2020.

## Built on by (most cited first, 17 of 17)

- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [Heterogeneous Acceleration Pipeline for Recommendation System Training](https://mlsanity.com/paper/2204.05436/) (2022)
- [Building a Performance Model for Deep Learning Recommendation Model Training on GPUs](https://mlsanity.com/paper/2201.07821/) (2022)
- [Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads](https://mlsanity.com/paper/2104.05755/) (2021)
- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)
- [Mem-Rec: Memory Efficient Recommendation System using Alternative Representation](https://mlsanity.com/paper/2305.07205/) (2023)
- [The trade-offs of model size in large recommendation models : A 10000 × compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)](https://mlsanity.com/paper/2207.10731/) (2022)
- [FPGA-based AI Smart NICs for Scalable Distributed AI Training Systems](https://mlsanity.com/paper/2204.10943/) (2022)
- [Deep Learning Models on CPUs: A Methodology for Efficient Training](https://mlsanity.com/paper/2206.10034/) (2022)
- [RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances](https://mlsanity.com/paper/2207.11434/) (2022)
- [PIFS-Rec: Process-In-Fabric-Switch for Large-Scale Recommendation System Inferences](https://mlsanity.com/paper/2409.16633/) (2024)
- [ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding](https://mlsanity.com/paper/2104.01981/) (2021)
- [Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations](https://mlsanity.com/paper/2302.11750/) (2023)
- [[Experiments & Analysis] Evaluating the Feasibility of Sampling-Based Techniques for Training Multilayer Perceptrons](https://mlsanity.com/paper/2306.09293/) (2023)
- [Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings](https://mlsanity.com/paper/2404.04270/) (2024)
- [Efficient Tabular Data Preprocessing of ML Pipelines](https://mlsanity.com/paper/2409.14912/) (2024)
- [Faster Distributed Inference-Only Recommender Systems via Bounded Lag Synchronous Collectives](https://mlsanity.com/paper/2512.19342/) (2025)

## Abstract (arXiv)

During the last two years, the goal of many researchers has been to squeeze the last bit of performance out of HPC system for AI tasks. Often this discussion is held in the context of how fast ResNet50 can be trained. Unfortunately, ResNet50 is no longer a representative workload in 2020. Thus, we focus on Recommender Systems which account for most of the AI cycles in cloud computing centers. More specifically, we focus on Facebook's DLRM benchmark. By enabling it to run on latest CPU hardware and software tailored for HPC, we are able to achieve more than two-orders of magnitude improvement in performance (110x) on a single socket compared to the reference CPU implementation, and high scaling efficiency up to 64 sockets, while fitting ultra-large datasets. This paper discusses the optimization techniques for the various operators in DLRM and which component of the systems are stressed by these different operators. The presented techniques are applicable to a broader set of DL workloads that pose the same scaling challenges/characteristics as DLRM.