mlsanityData to 7 Oct 2026

Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization

arXiv 2405.18890 ↗First posted 29 May 2024cs.LG cs.DCcited by 28

Not on the map. Why?

Abstract (arXiv)

In federated learning (FL), the multi-step update and data heterogeneity among clients often lead to a loss landscape with sharper minima, degenerating the performance of the resulted global model. Prevalent federated approaches incorporate sharpness-aware minimization (SAM) into local training to mitigate this problem. However, the local loss landscapes may not accurately reflect the flatness of global loss landscape in heterogeneous environments; as a result, minimizing local sharpness and calculating perturbations on client data might not align the efficacy of SAM in FL with centralized training. To overcome this challenge, we propose FedLESAM, a novel algorithm that locally estimates the direction of global perturbation on client side as the difference between global models received in the previous active and current rounds. Besides the improved quality, FedLESAM also speed up federated SAM-based approaches since it only performs once backpropagation in each iteration. Theoretically, we prove a slightly tighter bound than its original FedSAM by ensuring consistent perturbation. Empirically, we conduct comprehensive experiments on four federated benchmark datasets under three partition strategies to demonstrate the superior performance and efficiency of FedLESAM.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Group Normalizationcited by 1,514
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2020Adaptive Federated Optimizationcited by 1,036
  12. 2019Deep Leakage from Gradientscited by 1,004
29 more
  1. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  4. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  5. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  6. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  8. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  9. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  10. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  11. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  12. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  13. 2022Sharpness-Aware Training for Freecited by 95
  14. 2021Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learningcited by 78
  15. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  16. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  17. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  18. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  19. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  20. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  21. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 43
  22. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  23. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  24. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  25. 2023Long-Tailed Partial Label Learning via Dynamic Rebalancingcited by 22
  26. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  27. 2022FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregationcited by 16
  28. 2023Federated Learning without Full Labels: A Surveycited by 12
  29. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 7.

28 papers.

Sort:

By area:

  1. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  2. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  3. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  4. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  5. 2024Diversified Batch Selection for Training Accelerationcited by 14
  6. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  7. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  8. 2025Federated Domain Generalization with Data-free On-server Matching Gradientcited by 7
  9. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  10. 2025Federated Learning for Diffusion Modelscited by 6
  11. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  12. 2025Federated ADMM from Bayesian Dualitycited by 3
16 more
  1. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  2. 2025FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysiscited by 2
  3. 2025FedSDA: Federated Stain Distribution Alignment for Non-IID Histopathological Image Classificationcited by 2
  4. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learningcited by 1
  5. 2026Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimizationcited by 1
  6. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
  7. 2024DP2-FedSAM: Enhancing Differentially Private Federated Learning Through Personalized Sharpness-Aware Minimizationcited by 0
  8. 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarioscited by 0
  9. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  10. 2025LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimizationcited by 0
  11. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  12. 2025FedSCAM (Federated Sharpness-Aware Minimization with Clustered Aggregation and Modulation): Scam-resistant SAM for Robust Federated Optimization in Heterogeneous Environmentscited by 0
  13. 2026Beyond Fixed Rounds: Data-Free Early Stopping for Practical Federated Learningcited by 0
  14. 2026FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learningcited by 0
  15. 2026FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filteringcited by 0
  16. 2026SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneitycited by 0