mlsanityData to 7 Oct 2026

Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awareness

arXiv 2408.14144 ↗First posted 26 Aug 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) can be coordinated under the orchestration of a central server to collaboratively build a privacy-preserving model without the need for data exchange. However, participant data heterogeneity leads to local optima divergence, subsequently affecting convergence outcomes. Recent research has focused on global sharpness-aware minimization (SAM) and dynamic regularization techniques to enhance consistency between global and local generalization and optimization objectives. Nonetheless, the estimation of global SAM introduces additional computational and memory overhead, while dynamic regularization suffers from bias in the local and global dual variables due to training isolation. In this paper, we propose a novel FL algorithm, FedTOGA, designed to consider optimization and generalization objectives while maintaining minimal uplink communication overhead. By linking local perturbations to global updates, global generalization consistency is improved. Additionally, global updates are used to correct local dynamic regularizers, reducing dual variables bias and enhancing optimization consistency. Global updates are passively received by clients, reducing overhead. We also propose neighborhood perturbation to approximate local perturbation, analyzing its strengths and limitations. Theoretical analysis shows FedTOGA achieves faster convergence O(1/T) under non-convex functions. Empirical studies demonstrate that FedTOGA outperforms state-of-the-art algorithms, with a 1% accuracy increase and 30% faster convergence, achieving state-of-the-art.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

29 of 41 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  7. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  8. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  9. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  10. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  11. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  12. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
17 more
  1. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  2. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  3. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  4. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  5. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  6. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  7. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  8. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  9. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  10. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  11. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  12. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  13. 2022FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregationcited by 16
  14. 2023On the Role of Server Momentum in Federated Learningcited by 16
  15. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  16. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  17. 2024Federated Learning under Partially Class-Disjoint Data via Manifold Reshapingcited by 4

Built on by

Not cited yet.