# DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

arXiv: https://arxiv.org/abs/2602.19945 · HTML: https://mlsanity.com/paper/2602.19945/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models](https://mlsanity.com/paper/2402.19449/) (2024)
- [Differentially Private Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2111.09278/) (2021)
- [Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimization](https://mlsanity.com/paper/2404.04454/) (2024)
- [Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting](https://mlsanity.com/paper/2507.02939/) (2025)
- [Faster Adaptive Federated Learning](https://mlsanity.com/paper/2212.00974/) (2022)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [BALF: Simple and Efficient Blur Aware Local Feature Detector](https://mlsanity.com/paper/2211.14731/) (2022)
- [Make Landscape Flatter in Differentially Private Federated Learning](https://mlsanity.com/paper/2303.11242/) (2023)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity](https://mlsanity.com/paper/2405.16585/) (2024)
- [Toward Communication Efficient Adaptive Gradient Method](https://mlsanity.com/paper/2109.05109/) (2021)
- [FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging](https://mlsanity.com/paper/2507.20016/) (2025)
- [Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges](https://mlsanity.com/paper/2601.20913/) (2026)
- [Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup](https://mlsanity.com/paper/2308.00522/) (2023)
- [MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset](https://mlsanity.com/paper/2303.12756/) (2023)
- [CLIPCleaner: Cleaning Noisy Labels with CLIP](https://mlsanity.com/paper/2408.10012/) (2024)
- [FedMuon: Accelerating Federated Learning with Matrix Orthogonalization](https://mlsanity.com/paper/2510.27403/) (2025)

## Built on by (most cited first, 1 of 1)

- [Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2608.03277/) (2026)

## Abstract (arXiv)

Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter (varepsilon,δ)-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, varepsilon=1), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available in Appendix.