Partial Parameter Updates for Efficient Distributed Training
Not on the map. Why?
Abstract (arXiv)
We introduce a memory- and compute-efficient method for low-communication distributed training. Existing methods reduce communication by performing multiple local updates between infrequent global synchronizations. We demonstrate that their efficiency can be significantly improved by restricting backpropagation: instead of updating all the parameters, each node updates only a fixed subset while keeping the remainder frozen during local steps. This constraint substantially reduces peak memory usage and training FLOPs, while a full forward pass over all parameters eliminates the need for cross-node activation exchange. Experiments on a 1.3B-parameter language model trained across 32 nodes show that our method matches the perplexity of prior low-communication approaches under identical token and bandwidth budgets while reducing training FLOPs and peak memory.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
34 of 43 references.
References · 34
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2025Qwen3 Technical Reportcited by 8,820
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2025Gemma 3 Technical Reportcited by 1,824
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
22 more
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 20218-bit Optimizers via Block-wise Quantizationcited by 403
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
- 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
- 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
- 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
- 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
- 2021Decentralized Federated Averagingcited by 114
- 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
- 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
- 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
- 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
- 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
- 2023Memory Efficient Optimizers with 4-bit Statescited by 54
- 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
- 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
- 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
- 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
- 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
- 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
- 2025Eager Updates For Overlapped Communication and Computation in DiLoCocited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.