mlsanityData to 7 Oct 2026

ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training

arXiv 2406.02613 ↗First posted 3 Jun 2024cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel setups introduces communication overhead that grows with the number of workers, limiting parallelization efficiency. Local optimization algorithms reduce communications but incur high memory costs as they prevent optimizer state sharding, hindering scalability. To address this, we propose ACcumulate while COmmunicate (ACCO), a memory-efficient optimization algorithm for distributed LLM training. By synchronizing delayed gradients while computing new ones, ACCO reduces GPU idle time and supports heterogeneous hardware. To mitigate the convergence issues caused by delayed updates, we introduce a novel technique ensuring training dynamics align with standard distributed optimization. Compared to ZeRO-1, our approach is significantly faster and scales effectively across heterogeneous hardware.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 55 of its 79 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

55 of 79 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  12. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
38 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  3. 2019Asynchronous Federated Optimizationcited by 264
  4. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  5. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  6. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  7. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  8. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  9. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  10. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  11. 2018i-RevNet: Deep Invertible Networkscited by 137
  12. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  13. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  14. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  15. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  16. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  17. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  18. 2014Deep learning with Elastic Averaging SGDcited by 90
  19. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  20. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  21. 2021Distributed Deep Learning in Open Collaborationscited by 58
  22. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  23. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  24. 2018A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updatescited by 47
  25. 2023Reversible Vision Transformerscited by 33
  26. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  27. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  28. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  29. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  30. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  31. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  32. 2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
  33. 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
  34. 2017YellowFin and the Art of Momentum Tuningcited by 16
  35. 2021A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossipcited by 15
  36. 2019Fully Decoupled Neural Network Learning Using Delayed Gradientscited by 12
  37. 2020Pipelined Backpropagation at Scale: Training Large Models without Batchescited by 11
  38. 2022Delay-adaptive step-sizes for asynchronous learningcited by 11

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  2. 2025Generative AI in Transportation Planning: A Surveycited by 8
  3. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  4. 2026Heterogeneous Low-Bandwidth Pre-Training of LLMscited by 1
  5. 2026Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagationcited by 0
  6. 2026Learned Subspace Compression for Communication-Efficient Pipeline Parallelismcited by 0