mlsanityData to 14 Oct 2026

Nesterov Method for Asynchronous Pipeline Parallel Optimization

arXiv 2505.01099 ↗First posted 2 May 2025cs.LG cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

Pipeline Parallelism (PP) enables large neural network training on small, interconnected devices by splitting the model into multiple stages. To maximize pipeline utilization, asynchronous optimization is appealing as it offers 100% pipeline utilization by construction. However, it is inherently challenging as the weights and gradients are no longer synchronized, leading to stale (or delayed) gradients. To alleviate this, we introduce a variant of Nesterov Accelerated Gradient (NAG) for asynchronous optimization in PP. Specifically, we modify the look-ahead step in NAG to effectively address the staleness in gradients. We theoretically prove that our approach converges at a sublinear rate in the presence of fixed delay in gradients. Our experiments on large-scale language modelling tasks using decoder-only architectures with up to 1B parameters, demonstrate that our approach significantly outperforms existing asynchronous methods, even surpassing the synchronous baseline.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

22 of 40 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  8. 2014Convex Optimization: Algorithms and Complexitycited by 575
  9. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  10. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  11. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  12. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
10 more
  1. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  2. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  3. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  4. 2023Zero Bubble Pipeline Parallelismcited by 40
  5. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  6. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  7. 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
  8. 2019XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Trainingcited by 20
  9. 2019Gap Aware Mitigation of Gradient Stalenesscited by 14
  10. 2019Taming Momentum in a Distributed Asynchronous Environmentcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelismcited by 3
  2. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  3. 2026AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Trainingcited by 1
  4. 2026Unextractable Protocol Models: Collaborative Training and Inference without Weight Materializationcited by 0
  5. 2026Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistencycited by 0
  6. 2026One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretrainingcited by 0