mlsanityData to 14 Oct 2026

NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

arXiv 2604.06956 ↗First posted 8 Apr 2026cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to data movement, especially lookup and communication latency associated with embeddings. Existing solutions either optimize only one bottleneck or improve throughput by sacrificing training consistency. This paper presents NestPipe, a large-scale decentralized embedding training framework that tackles both bottlenecks while preserving synchronous training semantics. NestPipe exploits two hierarchical sparse parallelism opportunities through nested pipelining. At the inter-batch level, Dual-Buffer Pipelining (DBP) constructs a staleness-free five-stage pipeline through dual-buffer synchronization, mitigating lookup bottlenecks without embedding staleness. At the intra-batch level, we identify the embedding freezing phenomenon, which inspires Frozen-Window Pipelining (FWP) to overlap All2All communication with dense computation via coordinated stream scheduling and key-centric sample clustering. Experiments on production GPU and NPU clusters with 1,536 workers demonstrate that NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

33 of 50 references.

References · 33

  1. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
  2. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 86
  3. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  4. 2022KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videoscited by 72
  5. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  6. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 49
  7. 2025MTGR: Industrial-Scale Generative Recommendation Framework in Meituancited by 49
  8. 2023Scaling Law of Large Sequential Recommendation Modelscited by 23
  9. 2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
  10. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  11. 2022PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systemscited by 17
  12. 2022AutoShard: Automated Embedding Table Sharding for Recommender Systemscited by 17
21 more
  1. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  2. 2022Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwardscited by 11
  3. 2022BagPipe: Accelerating Deep Recommendation Model Trainingcited by 10
  4. 2025FuXi-α: Scaling Recommendation Model with Feature Interaction Enhanced Transformercited by 10
  5. 2025Climber: Toward Efficient Scaling Laws for Large Recommendation Modelscited by 8
  6. 2025TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inferencecited by 8
  7. 2024Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendationcited by 6
  8. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 6
  9. 2025Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systemscited by 6
  10. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  11. 2024Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compressioncited by 5
  12. 2022GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Modelcited by 4
  13. 2024Mixed-Precision Embeddings for Large-Scale Recommendation Modelscited by 4
  14. 2026Bending the Scaling Law Curve in Large-Scale Recommendation Systemscited by 4
  15. 2025The Evolution of Embedding Table Optimization and Multi-Epoch Training in Pinterest Ads Conversioncited by 3
  16. 2024DQRM: Deep Quantized Recommendation Modelscited by 2
  17. 2025Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Trainingcited by 2
  18. 2025RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Modelscited by 2
  19. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  20. 2026Revisiting Parameter Server in LLM Post-Trainingcited by 2
  21. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2

Built on by

Not cited yet.