mlsanityData to 14 Oct 2026

FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging

arXiv 2507.20016 ↗First posted 26 Jul 2025cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

For federated learning (FL) algorithms such as FedSAM, their generalization capability is crucial for real-word applications. In this paper, we revisit the generalization problem in FL and investigate the impact of data heterogeneity on FL generalization. We find that FedSAM usually performs worse than FedAvg in the case of highly heterogeneous data, and thus propose a novel and effective federated learning algorithm with Stochastic Weight Averaging (called textttFedSWA), which aims to find flatter minima in the setting of highly heterogeneous data. Moreover, we introduce a new momentum-based stochastic controlled weight averaging FL algorithm (textttFedMoSWA), which is designed to better align local and global models. Theoretically, we provide both convergence analysis and generalization bounds for textttFedSWA and textttFedMoSWA. We also prove that the optimization and generalization errors of textttFedMoSWA are smaller than those of their counterparts, including FedSAM and its variants. Empirically, experimental results on CIFAR10/100 and Tiny ImageNet demonstrate the superiority of the proposed algorithms compared to their counterparts. Open source code at: https://github.com/junkangLiu0/FedSWA.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

34 of 49 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2020The Future of Digital Health with Federated Learningcited by 633
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  12. 2014Convex Optimization: Algorithms and Complexitycited by 575
22 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  2. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  3. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  4. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  5. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  6. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  7. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  8. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  9. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  10. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  11. 2022When Do Flat Minima Optimizers Work?cited by 78
  12. 2020Differentially Private Secure Multi-Party Computation for Federated Learning in Financial Applicationscited by 58
  13. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  14. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  15. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  16. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  17. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 28
  18. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  19. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  20. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  21. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  22. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  2. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  3. 2025DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Normcited by 9
  4. 2025CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Trackingcited by 9
  5. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  6. 2026FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Trackingcited by 7
  7. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 6
  8. 2025CATCH: A Modular Cross-domain Adaptive Template with Hookcited by 6
  9. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  10. 2026Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Datacited by 4
  11. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
  12. 2025Contrastive Prompt Clustering for Weakly Supervised Semantic Segmentationcited by 1
13 more
  1. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  2. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  3. 2026Deconstructing the Failure of Ideal Noise Correction: A Three-Pillar Diagnosiscited by 1
  4. 2026A Step Toward Federated Pretraining of Multimodal Large Language Modelscited by 1
  5. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  6. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  7. 2025FRIEREN: Federated Learning with Vision-Language Regularization for Segmentationcited by 0
  8. 2025FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detectioncited by 0
  9. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  10. 2026Fusion and Alignment Enhancement with Large Language Models for Tail-item Sequential Recommendationcited by 0
  11. 2026CTP-FL: Common-Trajectory Gradient Prediction for Federated Learningcited by 0
  12. 2026FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Trainingcited by 0
  13. 2026FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projectioncited by 0