mlsanityData to 7 Oct 2026

Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

arXiv 2403.00877 ↗First posted 1 Mar 2024cs.LG cs.DC cs.IRcited by 6

Not on the map. Why?

Abstract (arXiv)

We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

22 of 49 references.

References · 22

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  7. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  8. 2018xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systemscited by 277
  9. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  10. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  11. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 62
  12. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 40
10 more
  1. 2020Learning to Embed Categorical Features without Embedding Tables for Recommendationcited by 38
  2. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 37
  3. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  4. 2022MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloudcited by 21
  5. 2018Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Trainingcited by 17
  6. 2023A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directionscited by 17
  7. 2022DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Predictioncited by 12
  8. 2022Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertisingcited by 5
  9. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  10. 2023Ad-Rec: Advanced Feature Interactions to Address Covariate-Shifts in Recommendation Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 3.

6 papers.

  1. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  2. 2024Wukong: Towards a Scaling Law for Large-Scale Recommendationcited by 72
  3. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  4. 2025Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimationcited by 0
  5. 2026NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipeliningcited by 0
  6. 2026ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendationcited by 0