mlsanityData to 14 Oct 2026

Private Adaptive Optimization with Side Information

arXiv 2202.05963 ↗First posted 12 Feb 2022cs.LG cs.CR stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Adaptive optimization methods have become the default solvers for many machine learning tasks. Unfortunately, the benefits of adaptivity may degrade when training with differential privacy, as the noise added to ensure privacy reduces the effectiveness of the adaptive preconditioner. To this end, we propose AdaDPS, a general framework that uses non-sensitive side information to precondition the gradients, allowing the effective use of adaptive methods in private settings. We formally show AdaDPS reduces the amount of noise needed to achieve similar privacy guarantees, thereby improving optimization performance. Empirically, we leverage simple and readily available side information to explore the performance of AdaDPS in practice, comparing to strong baselines in both centralized and federated settings. Our results show that AdaDPS improves accuracy by 7.7% (absolute) on average -- yielding state-of-the-art privacy-utility trade-offs on large-scale text and image benchmarks.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

18 of 36 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2021A Field Guide to Federated Optimizationcited by 315
  9. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  10. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  11. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  12. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
6 more
  1. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  2. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  3. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  4. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  5. 2015Heterogeneous Differential Privacycited by 30
  6. 2020Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Boundscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 13; 2024: 4; 2025: 8; 2026: 6.

38 papers.

Sort:

By area:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  2. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  3. 2023Selective Pre-training for Private Fine-tuningcited by 23
  4. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  5. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  6. 2022Membership Inference Attack Using Self Influence Functionscited by 20
  7. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  8. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  9. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  10. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  11. 2023PILLAR: How to make semi-private learning more effectivecited by 10
  12. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
26 more
  1. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  2. 2023Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imagingcited by 6
  3. 2023Private Learning with Public Featurescited by 6
  4. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  5. 2022Differentially Private Online-to-Batch for Smooth Lossescited by 5
  6. 2022Learning-Augmented Private Algorithms for Multiple Quantile Releasecited by 5
  7. 2023Efficient Federated Learning with Enhanced Privacy via Lottery Ticket Pruning in Edge Computingcited by 5
  8. 2024Delving into Differentially Private Transformercited by 5
  9. 2025Differentially Private Federated Learning With Time-Adaptive Privacy Spendingcited by 5
  10. 2023Private Matrix Factorization with Public Item Featurescited by 3
  11. 2023Coupling public and private gradient provably helps optimizationcited by 3
  12. 2025DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learningcited by 3
  13. 2024Provable Privacy with Non-Private Pre-Processingcited by 2
  14. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  15. 2025DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuningcited by 2
  16. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 2
  17. 2024Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weightcited by 1
  18. 2023AIJack: Let's Hijack AI! Security and Privacy Risk Simulator for Machine Learningcited by 0
  19. 2024Communication-Efficient and Privacy-Preserving Decentralized Meta-Learningcited by 0
  20. 2025Differentially Private Federated Clustering with Random Rebalancingcited by 0
  21. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  22. 2026Integrating Feature Correlation in Differential Privacy with Applications in DP-ERMcited by 0
  23. 2026DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learningcited by 0
  24. 2026PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRAcited by 0
  25. 2026Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetworkcited by 0
  26. 2026Differentially Private Natural Gradient Descentcited by 0