mlsanityData to 14 Oct 2026

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism

arXiv 2601.22442 ↗First posted 30 Jan 2026cs.LG cs.DCcited by 3

Not on the map. Why?

Abstract (arXiv)

Data and pipeline parallelism are key strategies for scaling neural network training across distributed devices, but their high communication cost necessitates co-located computing clusters with fast interconnects, limiting their scalability. We address this communication bottleneck by introducing asynchronous updates across both parallelism axes, relaxing the co-location requirement at the expense of introducing staleness between pipeline stages and data parallel replicas. To mitigate staleness, for pipeline parallelism, we adopt a weight look-ahead approach, and for data parallelism, we introduce an asynchronous sparse averaging method equipped with an exponential moving average based correction mechanism. We provide convergence guarantees for both sparse averaging and asynchronous updates. Experiments on large-scale language models (up to em 1B parameters) demonstrate that our approach matches the performance of the fully synchronous baseline, while significantly reducing communication overhead.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

41 of 63 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  11. 2020Adaptive Federated Optimizationcited by 1,046
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
29 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  3. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  5. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  6. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  7. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  8. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  9. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  10. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  11. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  12. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  13. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  14. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  15. 2023Zero Bubble Pipeline Parallelismcited by 40
  16. 2016Efficient Distributed Learning with Sparsitycited by 38
  17. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  18. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  19. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  20. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
  21. 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
  22. 2019XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Trainingcited by 20
  23. 2019Gap Aware Mitigation of Gradient Stalenesscited by 14
  24. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  25. 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 11
  26. 2019Taming Momentum in a Distributed Asynchronous Environmentcited by 9
  27. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  28. 2022Partial Model Averaging in Federated Learning: Performance Guarantees and Benefitscited by 8
  29. 2025Nesterov Method for Asynchronous Pipeline Parallel Optimizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averagingcited by 1
  2. 2026Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Modelscited by 0
  3. 2026Communication-Efficient LLM Adaptation over Decentralized GPU Meshescited by 0