mlsanityData to 7 Oct 2026

DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation

arXiv 2304.11208 ↗First posted 21 Apr 2023cs.LG cs.CRcited by 8

Not on the map. Why?

Abstract (arXiv)

We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 5 of its 7 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

5 of 7 references.

References · 5

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 3.

8 papers.

  1. 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
  2. 2024Revisiting Differentially Private Hyper-parameter Tuningcited by 5
  3. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 1
  4. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  5. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 1
  6. 2025Optimizer Dynamics at the Edge of Stability with Differential Privacycited by 0
  7. 2026Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theorycited by 0
  8. 2026Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Modelscited by 0