# Differentially Private Adaptive Optimization with Delayed Preconditioners

arXiv: https://arxiv.org/abs/2212.00309 · HTML: https://mlsanity.com/paper/2212.00309/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [R'enyi Differential Privacy of the Sampled Gaussian Mechanism](https://mlsanity.com/paper/1908.10530/) (2019)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [Practical and Private (Deep) Learning without Sampling or Shuffling](https://mlsanity.com/paper/2103.00039/) (2021)
- [Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification](https://mlsanity.com/paper/2007.03813/) (2020)
- [Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams](https://mlsanity.com/paper/2202.08312/) (2022)
- [Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization](https://mlsanity.com/paper/2010.09063/) (2020)
- [Online Adaptive Methods, Universality and Acceleration](https://mlsanity.com/paper/1809.02864/) (2018)
- [Public Data-Assisted Mirror Descent for Private Model Training](https://mlsanity.com/paper/2112.00193/) (2021)
- [Private Adaptive Gradient Methods for Convex Optimization](https://mlsanity.com/paper/2106.13756/) (2021)
- [Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration](https://mlsanity.com/paper/1807.06766/) (2018)
- [Private Adaptive Optimization with Side Information](https://mlsanity.com/paper/2202.05963/) (2022)
- [A new regret analysis for Adam-type algorithms](https://mlsanity.com/paper/2003.09729/) (2020)
- [Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds](https://mlsanity.com/paper/2006.13501/) (2020)

## Built on by (most cited first, 18 of 18)

- [DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)](https://mlsanity.com/paper/2312.14334/) (2023)
- [Practical Differentially Private Hyperparameter Tuning with Subsampling](https://mlsanity.com/paper/2301.11989/) (2023)
- [A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization](https://mlsanity.com/paper/2212.04486/) (2022)
- [Efficient Distributed Optimization under Heavy-Tailed Noise](https://mlsanity.com/paper/2502.04164/) (2025)
- [Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imaging](https://mlsanity.com/paper/2307.12542/) (2023)
- [On Design Principles for Private Adaptive Optimizers](https://mlsanity.com/paper/2507.01129/) (2025)
- [DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning](https://mlsanity.com/paper/2406.02744/) (2024)
- [Momentum Approximation in Asynchronous Private Federated Learning](https://mlsanity.com/paper/2402.09247/) (2024)
- [Efficient Adaptive Federated Optimization](https://mlsanity.com/paper/2410.18117/) (2024)
- [Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective](https://mlsanity.com/paper/2603.03226/) (2026)
- [Correlating Cross-Iteration Noise for DP-SGD using Model Curvature](https://mlsanity.com/paper/2510.05416/) (2025)
- [When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning](https://mlsanity.com/paper/2602.10584/) (2026)
- [Communication-Efficient and Privacy-Preserving Decentralized Meta-Learning](https://mlsanity.com/paper/2406.13183/) (2024)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [Deep Learning under Fractional-Order Differential Privacy](https://mlsanity.com/paper/2605.09890/) (2026)
- [DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning](https://mlsanity.com/paper/2605.13418/) (2026)
- [PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA](https://mlsanity.com/paper/2606.00944/) (2026)
- [Private Decentralized Optimization with Noise Reduction and Bias Correction](https://mlsanity.com/paper/2609.27658/) (2026)

## Abstract (arXiv)

Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training. Prior works typically address this issue by using auxiliary information (e.g., public data) to boost the effectiveness of adaptive optimization. In this work, we explore techniques to estimate and efficiently adapt to gradient geometry in private adaptive optimization without auxiliary data. Motivated by the observation that adaptive methods can tolerate stale preconditioners, we propose differentially private adaptive training with delayed preconditioners (DP^2), a simple method that constructs delayed but less noisy preconditioners to better realize the benefits of adaptivity. Theoretically, we provide convergence guarantees for our method for both convex and non-convex problems, and analyze trade-offs between delay and privacy noise reduction. Empirically, we explore DP^2 across several real-world datasets, demonstrating that it can improve convergence speed by as much as 4x relative to non-adaptive baselines and match the performance of state-of-the-art optimization methods that require auxiliary data.