Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
Not on the map. Why?
Abstract (arXiv)
Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays. This work considers a distributed training framework where each worker node is allowed to perform local model updates and the resulting models are averaged periodically. We analyze the true speed of error convergence with respect to wall-clock time (instead of the number of iterations), and analyze how it is affected by the frequency of averaging. The main contribution is the design of AdaComm, an adaptive communication strategy that starts with infrequent averaging to save communication delay and improve convergence speed, and then increases the communication frequency in order to achieve a low error floor. Rigorous experiments on training deep neural networks show that AdaComm can take 3 × less time than fully synchronous SGD, and still reach the same final training loss.
Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 24; 2022: 17; 2023: 12; 2024: 11; 2025: 7; 2026: 5.
76 papers.
Sort:
- 2021A Field Guide to Federated Optimizationcited by 312
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
- 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
- 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
- 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
- 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 35
- 2022Trading Off Privacy, Utility and Efficiency in Federated Learningcited by 33
- 2021Resource-Efficient Federated Learningcited by 26
- 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
- 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
- 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
38 more
- 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
- 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
- 2021Fast Federated Learning by Balancing Communication Trade-Offscited by 18
- 2021Adversarial training in communication constrained federated learningcited by 17
- 2021Distributed Learning Systems with First-order Methodscited by 17
- 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
- 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
- 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
- 2021A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guaranteecited by 16
- 2021LocalNewton: Reducing Communication Bottleneck for Distributed Learningcited by 15
- 2021Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGXcited by 14
- 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
- 2021Semi-Decentralized Federated Edge Learning with Data and Device Heterogeneitycited by 12
- 2022Federated Learning with Flexible Controlcited by 12
- 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 10
- 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
- 2021Coordinating Momenta for Cross-silo Federated Learningcited by 9
- 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
- 2024Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networkscited by 9
- 2021Critical Learning Periods in Federated Learningcited by 8
- 2022Partial Model Averaging in Federated Learning: Performance Guarantees and Benefitscited by 8
- 2023Time-sensitive Learning for Heterogeneous Federated Edge Intelligencecited by 8
- 2022Personalized Federated Learning with Communication Compressioncited by 6
- 2022Multi-Model Federated Learning with Provable Guaranteescited by 5
- 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
- 2023Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computingcited by 5
- 2024FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantizationcited by 5
- 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
- 2021RingFed: Reducing Communication Costs in Federated Learning on Non-IID Datacited by 4
- 2022On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruningcited by 4
- 2022Distributed Non-Convex Optimization with One-Bit Compressors on Heterogeneous Data: Efficient and Resilient Algorithmscited by 4
- 2023Faster Federated Learning with Decaying Number of Local SGD Stepscited by 4
- 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
- 2022The Role of Local Steps in Local SGDcited by 2
- 2022STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronizationcited by 2
- 2022Federated Hypergradient Descentcited by 2
- 2022CC-FedAvg: Computationally Customized Federated Averagingcited by 2
- 2023Distributed Extra-gradient with Optimal Complexity and Communication Guaranteescited by 2
50 most cited of 76 All 76 (JSON)
- 2026AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization29 Sep 2026
- 2026ML-for-ML6 Aug 2026
- 2026Robust Federated Learning Under Real-World Client Churn8 Jul 2026
- 2026Optimizing Stochastic Gradient Push under Broadcast Communications16 Apr 2026
- 2026FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation29 Jan 2026
- 2025Time-varying Mixing Matrix Design for Energy-efficient Decentralized Federated Learning30 Dec 2025
- 2025FedOAED: Federated On-Device Autoencoder Denoiser for Heterogeneous Data under Limited Client Availability19 Dec 2025
- 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning27 Jul 2025
- 2025Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning23 Jul 2025
- 2025TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments6 Jul 2025
- 2025FDA-Opt: Federated Fine-Tuning via Dynamic Update Schedules7 May 2025
- 2025Communication Optimization for Decentralized Learning atop Bandwidth-limited Edge Networks16 Apr 2025
38 more
- 2024EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models10 Dec 2024
- 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis11 Nov 2024
- 2024No Need to Talk: Asynchronous Mixture of Language Models4 Oct 2024
- 2024Overlay-based Decentralized Federated Learning in Bandwidth-limited Networks8 Aug 2024
- 2024FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization26 Jun 2024
- 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods20 Jun 2024
- 2024Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging31 May 2024
- 2024Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networks22 Apr 2024
- 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey9 Apr 2024
- 2024On the Convergence of Federated Learning Algorithms without Data Similarity29 Feb 2024
- 2024Energy-efficient Decentralized Learning via Graph Sparsification5 Jan 2024
- 2023Sparse Training for Federated Learning with Regularized Error Correction21 Dec 2023
- 2023ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation18 Nov 2023
- 2023Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing13 Nov 2023
- 2023Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling24 Oct 2023
- 2023A Quadratic Synchronization Rule for Distributed Deep Learning22 Oct 2023
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction12 Oct 2023
- 2023Distributed Extra-gradient with Optimal Complexity and Communication Guarantees17 Aug 2023
- 2023GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems13 Jun 2023
- 2023A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging6 Jun 2023
- 2023Faster Federated Learning with Decaying Number of Local SGD Steps16 May 2023
- 2023Balancing Privacy and Performance for Private Federated Learning Algorithms11 Apr 2023
- 2023Time-sensitive Learning for Heterogeneous Federated Edge Intelligence26 Jan 2023
- 2022CC-FedAvg: Computationally Customized Federated Averaging28 Dec 2022
- 2022Federated Learning with Flexible Control16 Dec 2022
- 2022Federated Hypergradient Descent3 Nov 2022
- 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity28 Oct 2022
- 2022STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization6 Oct 2022
- 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning3 Oct 2022
- 2022Distributed Non-Convex Optimization with One-Bit Compressors on Heterogeneous Data: Efficient and Resilient Algorithms3 Oct 2022
- 2022Personalized Federated Learning with Communication Compression12 Sep 2022
- 2022Trading Off Privacy, Utility and Efficiency in Federated Learning1 Sep 2022
- 2022Multi-Model Federated Learning with Provable Guarantees9 Jul 2022
- 2022Efficient Adaptive Federated Optimization of Federated Learning for IoT23 Jun 2022
- 2022A Unified Analysis of Federated Learning with Arbitrary Client Participation26 May 2022
- 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learning27 Apr 2022
- 2022The Role of Local Steps in Local SGD14 Mar 2022
- 2022Evaluation and Analysis of Different Aggregation and Hyperparameter Selection Methods for Federated Brain Tumor Segmentation16 Feb 2022
50 newest of 76 All 76 (JSON)
- 2021A Field Guide to Federated Optimizationcited by 312
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
- 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
- 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
- 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
- 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 35
- 2022Trading Off Privacy, Utility and Efficiency in Federated Learningcited by 33
- 2021Resource-Efficient Federated Learningcited by 26
- 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
- 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
- 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
- 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
12 most cited of 76 All 76 (JSON)
- 2026AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization29 Sep 2026
- 2026ML-for-ML6 Aug 2026
- 2026Robust Federated Learning Under Real-World Client Churn8 Jul 2026
- 2026Optimizing Stochastic Gradient Push under Broadcast Communications16 Apr 2026
- 2026FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation29 Jan 2026
- 2025Time-varying Mixing Matrix Design for Energy-efficient Decentralized Federated Learning30 Dec 2025
- 2025FedOAED: Federated On-Device Autoencoder Denoiser for Heterogeneous Data under Limited Client Availability19 Dec 2025
- 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning27 Jul 2025
- 2025Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning23 Jul 2025
- 2025TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments6 Jul 2025
- 2025FDA-Opt: Federated Fine-Tuning via Dynamic Update Schedules7 May 2025
- 2025Communication Optimization for Decentralized Learning atop Bandwidth-limited Edge Networks16 Apr 2025
12 newest of 76 All 76 (JSON)