mlsanityData to 14 Oct 2026

Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization

arXiv 2306.05706 ↗First posted 9 Jun 2023cs.LG cs.DC math.OCcited by 21

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the ``client-drift'' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of the ``client drift'' and explore its substance in FL, in this paper, we first design an efficient FL algorithm textitFedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, textitFedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. This relaxed initialization helps to revise the local divergence and enhance the local consistency level. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error of the proposed textitFedInit method. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound in textitFedInit. Extensive experiments are conducted to validate this conclusion. Our proposed textitFedInit could achieve state-of-the-art~(SOTA) results compared to several advanced benchmarks without any additional costs. Meanwhile, stage-wise relaxed initialization could also be incorporated into the current advanced algorithms to achieve higher performance in the FL paradigm.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

45 of 53 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  12. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
33 more
  1. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  2. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  3. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  4. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  5. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  6. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  7. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  8. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  9. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  10. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  11. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  12. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  13. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  14. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  15. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  16. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  17. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  18. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  19. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  20. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  21. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  22. 2022Federated Learning via Inexact ADMMcited by 44
  23. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
  24. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  25. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  26. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  27. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  28. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25
  29. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 21
  30. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  31. 2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
  32. 2023Enhance Local Consistency in Federated Learning: A Multi-Step Inertial Momentum Approachcited by 11
  33. 2021Layer-wise and Dimension-wise Locally Adaptive Federated Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 5; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  2. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  3. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  4. 2024A Unified Analysis for Finite Weight Averagingcited by 3
  5. 2023Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variancecited by 2
  6. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  7. 2024Boosting the Performance of Decentralized Federated Learning via Catalyst Accelerationcited by 2
  8. 2024FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Datacited by 2
  9. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environmentscited by 2
  10. 2025Unveiling the Power of Multiple Gossip Steps: A Stability-Based Generalization Analysis in Decentralized Trainingcited by 2
  11. 2024OledFL: Unleashing the Potential of Decentralized Federated Learning via Opposite Lookahead Enhancementcited by 1
  12. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
9 more
  1. 2024Convergent Differential Privacy Analysis for General Federated Learningcited by 0
  2. 2024Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimizationcited by 0
  3. 2025Accelerated Training of Federated Learning via Second-Order Methodscited by 0
  4. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  5. 2026FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clientscited by 0
  6. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0
  7. 2026Cooperate to Compete: Strategic Data Generation and Incentivization Framework for Coopetitive Cross-Silo Federated Learningcited by 0
  8. 2026Learning Theory of the SVRG: Generalization and Convergence Analysiscited by 0
  9. 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learningcited by 0