mlsanityData to 7 Oct 2026

From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization

arXiv 2112.09355 ↗First posted 17 Dec 2021cs.DC cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In the setting of federated optimization, where a global model is aggregated periodically, step asynchronism occurs when participants conduct model training by efficiently utilizing their computational resources. It is well acknowledged that step asynchronism leads to objective inconsistency under non-i.i.d. data, which degrades the model's accuracy. To address this issue, we propose a new algorithm FedaGrac, which calibrates the local direction to a predictive global orientation. Taking advantage of the estimated orientation, we guarantee that the aggregated model does not excessively deviate from the global optimum while fully utilizing the local updates of faster nodes. We theoretically prove that FedaGrac holds an improved order of convergence rate than the state-of-the-art approaches and eliminates the negative effect of step asynchronism. Empirical results show that our algorithm accelerates the training and enhances the final accuracy.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 68 references that are in the map. First posted on arXiv 17 Dec 2021.

Builds on

37 of 68 references.

References · 37

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2018Federated Learning with Non-IID Datacited by 1,099
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  12. 2020Federated Learning with Matched Averagingcited by 564
25 more
  1. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  2. 2018Sparsified SGD with Memorycited by 373
  3. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  4. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  5. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  6. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  7. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 193
  8. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  9. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  10. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  11. 2020FedVision: An Online Visual Object Detection Platform Powered by Federated Learningcited by 121
  12. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  13. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  14. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  15. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  16. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  17. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 75
  18. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  19. 2020An Efficiency-boosting Client Selection Scheme for Federated Learning with Fairness Guaranteecited by 61
  20. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  21. 2019Federated Learning of N-gram Language Modelscited by 33
  22. 2020Communication-Efficient Federated Learning with Compensated Overlap-FedAvgcited by 31
  23. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  24. 2021Personalized Federated Learning with Contextualized Generalizationcited by 23
  25. 2020Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systemscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 3; 2025: 1; 2026: 2.

8 papers.

  1. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  2. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  3. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  4. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  5. 2024Hierarchical Federated Learning with Multi-Timescale Gradient Correctioncited by 7
  6. 2025Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Maskingcited by 0
  7. 2026Representation-Aligned Multi-Scale Personalization for Federated Learningcited by 0
  8. 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learningcited by 0