mlsanityData to 13 Oct 2026

Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization

arXiv 2604.12768 ↗First posted 14 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the ``client-drift'' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of ``client drift'' and explore its substance in FL, in this paper, we first propose an efficient FL algorithm FedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error in FL. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound. Extensive experiments are conducted to validate its efficiency. The proposed FedInit method could achieve comparable results compared to several advanced benchmarks without any additional training or communication costs. Meanwhile, the stage-wise personalized relaxed initialization could also be incorporated into several current advanced algorithms to achieve higher generalization performance in the FL paradigm.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 53 of its 61 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

53 of 61 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  4. 2019Advances and Open Problems in Federated Learningcited by 3,040
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  6. 2020Adaptive Federated Optimizationcited by 1,041
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 903
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 846
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 559
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 549
  12. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
38 more
  1. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  2. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 312
  3. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 304
  4. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  5. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  6. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  7. 2020The Lipschitz Constant of Self-Attentioncited by 172
  8. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  9. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  10. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  11. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  12. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  13. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  14. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  15. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  16. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  17. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  18. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  19. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  20. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  21. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  22. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  23. 2022Federated Learning via Inexact ADMMcited by 44
  24. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  25. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 35
  26. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  27. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  28. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  29. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  30. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25
  31. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  32. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 21
  33. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  34. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  35. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  36. 2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
  37. 2022Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradientscited by 11
  38. 2023Enhance Local Consistency in Federated Learning: A Multi-Step Inertial Momentum Approachcited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.