mlsanityData to 7 Oct 2026

DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

arXiv 2602.19945 ↗First posted 23 Feb 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter (varepsilon,δ)-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, varepsilon=1), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available in Appendix.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

43 of 75 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2019On the Convergence of Adam and Beyondcited by 834
  12. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
31 more
  1. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  3. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  4. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  5. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 53
  6. 2022Faster Adaptive Federated Learningcited by 35
  7. 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
  8. 2022BALF: Simple and Efficient Blur Aware Local Feature Detectorcited by 34
  9. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  10. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  11. 2024Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversitycited by 28
  12. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  13. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  14. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  15. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  16. 2023MaskCon: Masked Contrastive Learning for Coarse-Labelled Datasetcited by 21
  17. 2024CLIPCleaner: Cleaning Noisy Labels with CLIPcited by 16
  18. 2025Convex Relaxation for Robust Vanishing Point Estimation in Manhattan Worldcited by 15
  19. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  20. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  21. 2024LAFS: Landmark-based Facial Self-supervised Learning for Face Recognitioncited by 12
  22. 2026Advances in Global Solvers for 3D Visioncited by 12
  23. 2024PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attackscited by 11
  24. 2025DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Normcited by 9
  25. 2020Differentially Private Federated Learning with Laplacian Smoothingcited by 8
  26. 2024Cross-silo Federated Learning with Record-level Personalized Differential Privacycited by 7
  27. 2023DP-BREM: Differentially-Private and Byzantine-Robust Federated Learning with Client Momentumcited by 6
  28. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  29. 2025Dual Defense: Enhancing Privacy and Mitigating Poisoning Attacks in Federated Learningcited by 3
  30. 2025From Risk to Resilience: Towards Assessing and Mitigating the Risk of Data Reconstruction Attacks in Federated Learningcited by 3
  31. 2022Refiner: Data Refining against Gradient Leakage Attacks in Federated Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Modelscited by 0