# Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

arXiv: https://arxiv.org/abs/2403.00877 · HTML: https://mlsanity.com/paper/2403.00877/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [Scaling Distributed Machine Learning with In-Network Aggregation](https://mlsanity.com/paper/1903.06701/) (2019)
- [FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference](https://mlsanity.com/paper/2101.05615/) (2021)
- [Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems](https://mlsanity.com/paper/2003.09518/) (2020)
- [Learning to Embed Categorical Features without Embedding Tables for Recommendation](https://mlsanity.com/paper/2010.10784/) (2020)
- [Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms](https://mlsanity.com/paper/2007.00156/) (2020)
- [MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud](https://mlsanity.com/paper/2205.00119/) (2022)
- [Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training](https://mlsanity.com/paper/1805.07891/) (2018)
- [A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions](https://mlsanity.com/paper/2302.04473/) (2023)
- [DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction](https://mlsanity.com/paper/2203.11014/) (2022)
- [Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising](https://mlsanity.com/paper/2201.05500/) (2022)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [Ad-Rec: Advanced Feature Interactions to Address Covariate-Shifts in Recommendation Networks](https://mlsanity.com/paper/2308.14902/) (2023)

## Built on by (most cited first, 6 of 6)

- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://mlsanity.com/paper/2411.04996/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [LoKA: Low-precision Kernel Applications for Recommendation Models At Scale](https://mlsanity.com/paper/2605.10886/) (2026)
- [Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimation](https://mlsanity.com/paper/2505.01616/) (2025)
- [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](https://mlsanity.com/paper/2604.06956/) (2026)
- [ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendation](https://mlsanity.com/paper/2607.27744/) (2026)

## Abstract (arXiv)

We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.