mlsanityData to 7 Oct 2026

Improving Generalization in Federated Learning by Seeking Flat Minima

arXiv 2203.11834 ↗First posted 22 Mar 2022cs.LG cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

Models trained in federated settings often suffer from degraded performances and fail at generalizing, especially when facing heterogeneous scenarios. In this work, we investigate such behavior through the lens of geometry of the loss and Hessian eigenspectrum, linking the model's lack of generalization capacity to the sharpness of the solution. Motivated by prior studies connecting the sharpness of the loss surface and the generalization gap, we show that i) training clients locally with Sharpness-Aware Minimization (SAM) or its adaptive version (ASAM) and ii) averaging stochastic weights (SWA) on the server-side can substantially improve generalization in Federated Learning and help bridging the gap with centralized models. By seeking parameters in neighborhoods having uniform low loss, the model converges towards flatter minima and its generalization significantly improves in both homogeneous and heterogeneous scenarios. Empirical results demonstrate the effectiveness of those optimizers across a variety of benchmark vision datasets (e.g. CIFAR10/100, Landmarks-User-160k, IDDA) and tasks (large scale classification, semantic segmentation, domain generalization).

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 68 of its 93 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

68 of 93 references.

References · 68

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2019Advances and Open Problems in Federated Learningcited by 3,021
38 more
  1. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2018Group Normalizationcited by 1,514
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2018Federated Learning with Non-IID Datacited by 1,099
  12. 2020Adaptive Federated Optimizationcited by 1,036
  13. 2016An overview of gradient descent optimization algorithmscited by 1,021
  14. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  15. 2021Model-Contrastive Federated Learningcited by 762
  16. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  17. 2017Federated Multi-Task Learningcited by 669
  18. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  19. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  20. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  21. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  22. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  23. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  24. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  25. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  26. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  27. 2020BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentationcited by 294
  28. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  29. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  30. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  31. 2019Adversarial Examples Improve Image Recognitioncited by 283
  32. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  33. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
  34. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  35. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  36. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  37. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  38. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 30; 2024: 31; 2025: 23; 2026: 10.

98 papers.

Sort:

By area:

  1. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  2. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  3. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  4. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  5. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  6. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  7. 2022Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learningcited by 35
  8. 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
  9. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  10. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
  11. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  12. 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
38 more
  1. 2024Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metricscited by 25
  2. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  3. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  4. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  5. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  6. 2023Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombinationcited by 13
  7. 2023Flat Seeking Bayesian Neural Networkscited by 10
  8. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  9. 2023Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofingcited by 9
  10. 2022Semantic Self-adaptation: Enhancing Generalization with a Single Samplecited by 8
  11. 2023Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Trainingcited by 8
  12. 2024Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learningcited by 8
  13. 2023Federated Survival Forestscited by 7
  14. 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
  15. 2023Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learningcited by 7
  16. 2024Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfacescited by 7
  17. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  18. 2023Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory Distillationcited by 6
  19. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  20. 2023Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacycited by 5
  21. 2024From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matchingcited by 5
  22. 2024FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learningcited by 5
  23. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  24. 2023Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonizationcited by 4
  25. 2024Feature Diversification and Adaptation for Federated Domain Generalizationcited by 4
  26. 2025Interaction-Aware Gaussian Weighting for Clustered Federated Learningcited by 4
  27. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  28. 2023Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarioscited by 3
  29. 2023UPFL: Unsupervised Personalized Federated Learning towards New Clientscited by 3
  30. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  31. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  32. 2024Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentationcited by 3
  33. 2024Federated Adversarial Learning for Robust Autonomous Landing Runway Detectioncited by 3
  34. 2024An Uncertainty-Aware Generalization Framework for Cardiovascular Image Segmentationcited by 3
  35. 2025Understanding Flatness in Generative Models: Its Role and Benefitscited by 3
  36. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3
  37. 2025Federated EndoViT: Pretraining Vision Transformers via Federated Learning on Endoscopic Image Collectionscited by 3
  38. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3

50 most cited of 98 All 98 (JSON)