# NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

arXiv: https://arxiv.org/abs/2604.06956 · HTML: https://mlsanity.com/paper/2604.06956/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 30 of 33)

- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos](https://mlsanity.com/paper/2208.08696/) (2022)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](https://mlsanity.com/paper/2502.19811/) (2025)
- [MTGR: Industrial-Scale Generative Recommendation Framework in Meituan](https://mlsanity.com/paper/2505.18654/) (2025)
- [Scaling Law of Large Sequential Recommendation Models](https://mlsanity.com/paper/2311.11351/) (2023)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters](https://mlsanity.com/paper/2111.05897/) (2021)
- [PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems](https://mlsanity.com/paper/2204.04903/) (2022)
- [AutoShard: Automated Embedding Table Sharding for Recommender Systems](https://mlsanity.com/paper/2208.06399/) (2022)
- [Heterogeneous Acceleration Pipeline for Recommendation System Training](https://mlsanity.com/paper/2204.05436/) (2022)
- [Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards](https://mlsanity.com/paper/2205.04702/) (2022)
- [BagPipe: Accelerating Deep Recommendation Model Training](https://mlsanity.com/paper/2202.12429/) (2022)
- [FuXi-α: Scaling Recommendation Model with Feature Interaction Enhanced Transformer](https://mlsanity.com/paper/2502.03036/) (2025)
- [Climber: Toward Efficient Scaling Laws for Large Recommendation Models](https://mlsanity.com/paper/2502.09888/) (2025)
- [TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference](https://mlsanity.com/paper/2506.10470/) (2025)
- [Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation](https://mlsanity.com/paper/2403.00877/) (2024)
- [SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training](https://mlsanity.com/paper/2504.14519/) (2025)
- [Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems](https://mlsanity.com/paper/2506.17551/) (2025)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression](https://mlsanity.com/paper/2407.04272/) (2024)
- [GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model](https://mlsanity.com/paper/2205.11048/) (2022)
- [Mixed-Precision Embeddings for Large-Scale Recommendation Models](https://mlsanity.com/paper/2409.20305/) (2024)
- [Bending the Scaling Law Curve in Large-Scale Recommendation Systems](https://mlsanity.com/paper/2602.16986/) (2026)
- [The Evolution of Embedding Table Optimization and Multi-Epoch Training in Pinterest Ads Conversion](https://mlsanity.com/paper/2505.05605/) (2025)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)
- [Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training](https://mlsanity.com/paper/2508.03854/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)

## Abstract (arXiv)

Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to data movement, especially lookup and communication latency associated with embeddings. Existing solutions either optimize only one bottleneck or improve throughput by sacrificing training consistency. This paper presents NestPipe, a large-scale decentralized embedding training framework that tackles both bottlenecks while preserving synchronous training semantics. NestPipe exploits two hierarchical sparse parallelism opportunities through nested pipelining. At the inter-batch level, Dual-Buffer Pipelining (DBP) constructs a staleness-free five-stage pipeline through dual-buffer synchronization, mitigating lookup bottlenecks without embedding staleness. At the intra-batch level, we identify the embedding freezing phenomenon, which inspires Frozen-Window Pipelining (FWP) to overlap All2All communication with dense computation via coordinated stream scheduling and key-centric sample clustering. Experiments on production GPU and NPU clusters with 1,536 workers demonstrate that NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency.