# DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation

arXiv: https://arxiv.org/abs/2304.11208 · HTML: https://mlsanity.com/paper/2304.11208/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)

## Built on by (most cited first, 8 of 8)

- [DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)](https://mlsanity.com/paper/2312.14334/) (2023)
- [Revisiting Differentially Private Hyper-parameter Tuning](https://mlsanity.com/paper/2402.13087/) (2024)
- [Improved Algorithms for Differentially Private Language Model Alignment](https://mlsanity.com/paper/2505.08849/) (2025)
- [Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective](https://mlsanity.com/paper/2603.03226/) (2026)
- [Cocoon: A System Architecture for Differentially Private Training with Correlated Noises](https://mlsanity.com/paper/2510.07304/) (2025)
- [Optimizer Dynamics at the Edge of Stability with Differential Privacy](https://mlsanity.com/paper/2512.19019/) (2025)
- [Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theory](https://mlsanity.com/paper/2602.23516/) (2026)
- [Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2608.03277/) (2026)

## Abstract (arXiv)

We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.