mlsanityData to 7 Oct 2026

FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models

arXiv 2510.27486 ↗First posted 31 Oct 2025cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

AdamW has become one of the most effective optimizers for training large-scale models. We have also observed its effectiveness in the context of federated learning (FL). However, directly applying AdamW in federated learning settings poses significant challenges: (1) due to data heterogeneity, AdamW often yields high variance in the second-moment estimate boldsymbolv; (2) the local overfitting of AdamW may cause client drift; and (3) Reinitializing moment estimates (boldsymbolv, boldsymbolm) at each round slows down convergence. To address these challenges, we propose the first underlineFederated underlineAdamW algorithm, called textttFedAdamW, for training and fine-tuning various large models. textttFedAdamW aligns local updates with the global update using both a local correction mechanism and decoupled weight decay to mitigate local overfitting. textttFedAdamW efficiently aggregates the textttmean of the second-moment estimates to reduce their variance and reinitialize them. Theoretically, we prove that textttFedAdamW achieves a linear speedup convergence rate of O(√(L Delta σl2)/(S K R ε2)+(L Delta)/R) without heterogeneity assumption, where S is the number of participating clients per round, K is the number of local iterations, and R is the total number of communication rounds. We also employ PAC-Bayesian generalization analysis to explain the effectiveness of decoupled weight decay in local training. Empirically, we validate the effectiveness of textttFedAdamW on language and vision Transformer models. Compared to several baselines, textttFedAdamW significantly reduces communication rounds and improves test accuracy. The code is available in https://github.com/junkangLiu0/FedAdamW.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 41 of its 94 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

41 of 94 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2020Adaptive Federated Optimizationcited by 1,036
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
29 more
  1. 2019On the Convergence of Adam and Beyondcited by 834
  2. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  3. 2017Three Factors Influencing Minima in SGDcited by 270
  4. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  6. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  7. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  9. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  10. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  11. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  12. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  13. 2025FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrievalcited by 41
  14. 2025OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrievalcited by 40
  15. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  16. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  17. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  18. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 27
  19. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  20. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  21. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  22. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  23. 2025Convex Relaxation for Robust Vanishing Point Estimation in Manhattan Worldcited by 15
  24. 2024Information Entropy Guided Height-aware Histogram for Quantization-friendly Pillar Feature Encodercited by 14
  25. 2024GlobalPointer: Large-Scale Plane Adjustment with Bi-Convex Relaxationcited by 11
  26. 2022From a Bird's Eye View to See: Joint Camera and Subject Registration without the Camera Calibrationcited by 5
  27. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 5
  28. 2025Graph Learning Metallic Glass Discovery from Wikipediacited by 2
  29. 2025Dual Enhancement on 3D Vision-Language Perception for Monocular 3D Visual Groundingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 14.

15 papers.

Sort:

By area:

  1. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  2. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  3. 2026FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learningcited by 7
  4. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  5. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  6. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  7. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  8. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  9. 2026Fusion and Alignment Enhancement with Large Language Models for Tail-item Sequential Recommendationcited by 0
  10. 2026From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneitycited by 0
  11. 2026Personalized Federated Learning for Gradient Alignmentcited by 0
  12. 2026FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneitycited by 0
3 more
  1. 2026CTP-FL: Common-Trajectory Gradient Prediction for Federated Learningcited by 0
  2. 2026FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Trainingcited by 0
  3. 2026FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projectioncited by 0