mlsanityData to 7 Oct 2026

DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)

arXiv 2312.14334 ↗First posted 21 Dec 2023cs.LG cs.CRcited by 28

Not on the map. Why?

Abstract (arXiv)

The Adam optimizer is a popular choice in contemporary deep learning, due to its strong empirical performance. However we observe that in privacy sensitive scenarios, the traditional use of Differential Privacy (DP) with the Adam optimizer leads to sub-optimal performance on several tasks. We find that this performance degradation is due to a DP bias in Adam's second moment estimator, introduced by the addition of independent noise in the gradient computation to enforce DP guarantees. This DP bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam. We propose DP-AdamBC, an optimization algorithm which removes the bias in the second moment estimation and retrieves the expected behaviour of Adam. Empirically, DP-AdamBC significantly improves the optimization performance of DP-Adam by up to 3.5% in final accuracy in image, text, and graph node classification tasks.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

21 of 31 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  9. 2021Membership Inference Attacks From First Principlescited by 839
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  12. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
9 more
  1. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  2. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  3. 2022Reconstructing Training Data with Informed Adversariescited by 143
  4. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  5. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  6. 2021Node-Level Differentially Private Graph Neural Networkscited by 43
  7. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  8. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  9. 2023DP-Adam: Correcting DP Bias in Adam's Second Moment Estimationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 12.

28 papers.

Sort:

By area:

  1. 2024DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reductioncited by 10
  2. 2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8
  3. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  4. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  5. 2025Continual Release Moment Estimation with Differential Privacycited by 5
  6. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  7. 2025DP-Adam-AC: Privacy-preserving Fine-Tuning of Localizable Language Models Using Adam Optimization with Adaptive Clippingcited by 3
  8. 2025DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learningcited by 3
  9. 2025Accelerating Differentially Private Federated Learning via Adaptive Extrapolationcited by 1
  10. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 1
  11. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  12. 2025DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuningcited by 1
16 more
  1. 2026When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learningcited by 1
  2. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 1
  3. 2026FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correctioncited by 1
  4. 2026DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentumcited by 1
  5. 2026DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentumcited by 1
  6. 2024DP-KAN: Differentially Private Kolmogorov-Arnold Networkscited by 0
  7. 2025Fast Fourier Transform-Based Spectral and Temporal Gradient Filtering for Differential Privacycited by 0
  8. 2025On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalancecited by 0
  9. 2025Almost Sure Convergence Analysis of Differentially Private Stochastic Gradient Methodscited by 0
  10. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  11. 2026Deep Learning under Fractional-Order Differential Privacycited by 0
  12. 2026DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learningcited by 0
  13. 2026From Privacy to Generalization: Linear Max-Information Bounds for Differentially Private Learning Algorithmscited by 0
  14. 2026PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRAcited by 0
  15. 2026Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Modelscited by 0
  16. 2026Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noisecited by 0