mlsanityData to 14 Oct 2026

Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape

arXiv 2305.11584 ↗First posted 19 May 2023cs.LG cs.DC math.OCcited by 46

Not on the map. Why?

Abstract (arXiv)

In federated learning (FL), a cluster of local clients are chaired under the coordination of the global server and cooperatively train one model with privacy protection. Due to the multiple local updates and the isolated non-iid dataset, clients are prone to overfit into their own optima, which extremely deviates from the global objective and significantly undermines the performance. Most previous works only focus on enhancing the consistency between the local and global objectives to alleviate this prejudicial client drifts from the perspective of the optimization view, whose performance would be prominently deteriorated on the high heterogeneity. In this work, we propose a novel and general algorithm ttfamily FedSMOO by jointly considering the optimization and generalization targets to efficiently improve the performance in FL. Concretely, ttfamily FedSMOO adopts a dynamic regularizer to guarantee the local optima towards the global objective, which is meanwhile revised by the global Sharpness Aware Minimization (SAM) optimizer to search for the consistent flat minima. Our theoretical analysis indicates that ttfamily FedSMOO achieves fast O(1/T) convergence rate with low generalization bound. Extensive numerical studies are conducted on the real-world dataset to verify its peerless efficiency and excellent generality.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

49 of 60 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2018Group Normalizationcited by 1,524
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 605
37 more
  1. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  2. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  3. 2021Towards Personalized Federated Learningcited by 464
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  5. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  6. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  7. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  8. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  9. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  10. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  11. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  12. 2020Survey of Personalization Techniques for Federated Learningcited by 145
  13. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  14. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  15. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  16. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  17. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  18. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  19. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  20. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  21. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  22. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  23. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  24. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 66
  25. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  26. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  27. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  28. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  29. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  30. 2020Stochastic Client Selection for Federated Learning with Volatile Clientscited by 39
  31. 2014Empirical risk minimization for heavy-tailed lossescited by 37
  32. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
  33. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  34. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  35. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  36. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  37. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 17; 2025: 13; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  2. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 29
  3. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  4. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  5. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  6. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  7. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  8. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  9. 2024Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learningcited by 8
  10. 2023Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learningcited by 7
  11. 2025Federated Domain Generalization with Data-free On-server Matching Gradientcited by 7
  12. 2025Federated Learning for Diffusion Modelscited by 7
34 more
  1. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  2. 2024Lightweight Unsupervised Federated Learning with Pretrained Vision Language Modelcited by 5
  3. 2024From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matchingcited by 5
  4. 2025Revisit the Stability of Vanilla Federated Learning Under Diverse Conditionscited by 4
  5. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  6. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  7. 2025Understanding Federated Learning from IID to Non-IID dataset: An Experimental Studycited by 3
  8. 2025A Unified Benchmark of Federated Learning with Kolmogorov-Arnold Networks for Medical Imagingcited by 3
  9. 2025Debunking Optimization Myths in Federated Learning for Medical Image Classificationcited by 3
  10. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  11. 2024A Method for Enhancing Generalization of Adam by Multiple Integrationscited by 2
  12. 2025Aggregation on Learnable Manifolds for Asynchronous Federated Optimizationcited by 1
  13. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  14. 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarioscited by 1
  15. 2025When to Stop Federated Learning: Zero-Shot Generation of Synthetic Validation Data with Generative AI for Early Stoppingcited by 1
  16. 2026Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimizationcited by 1
  17. 2024FedGuCci: Making Local Models More Connected in Landscape for Federated Learningcited by 0
  18. 2024SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Datacited by 0
  19. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
  20. 2024Convergent Differential Privacy Analysis for General Federated Learningcited by 0
  21. 2024DP2-FedSAM: Enhancing Differentially Private Federated Learning Through Personalized Sharpness-Aware Minimizationcited by 0
  22. 2024Federated Learning with Relative Fairnesscited by 0
  23. 2024Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimizationcited by 0
  24. 2025Fair Federated Medical Image Classification Against Quality Shift via Inter-Client Progressive State Matchingcited by 0
  25. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  26. 2025LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimizationcited by 0
  27. 2026Beyond Fixed Rounds: Data-Free Early Stopping for Practical Federated Learningcited by 0
  28. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0
  29. 2026FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learningcited by 0
  30. 2026FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filteringcited by 0
  31. 2026SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneitycited by 0
  32. 2026A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimizationcited by 0
  33. 2026How Much Regularization Survives Averaging? Update Masking in Federated Learningcited by 0
  34. 2026Joint Optimization for Federated Learning and Transmission over Unreliable Wireless Networks with Heterogeneous Datacited by 0