mlsanityData to 7 Oct 2026

DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

arXiv 2403.02571 ↗First posted 5 Mar 2024cs.LG cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent developments have underscored the critical role of textitdifferential privacy (DP) in safeguarding individual data for training machine learning models. However, integrating DP oftentimes incurs significant model performance degradation due to the perturbation introduced into the training process, presenting a formidable challenge in the differentially private machine learning (DPML) field. To this end, several mitigative efforts have been proposed, typically revolving around formulating new DPML algorithms or relaxing DP definitions to harmonize with distinct contexts. In spite of these initiatives, the diminishment induced by DP on models, particularly large-scale models, remains substantial and thus, necessitates an innovative solution that adeptly circumnavigates the consequential impairment of model utility. In response, we introduce DPAdapter, a pioneering technique designed to amplify the model performance of DPML algorithms by enhancing parameter robustness. The fundamental intuition behind this strategy is that models with robust parameters are inherently more resistant to the noise introduced by DP, thereby retaining better performance despite the perturbations. DPAdapter modifies and enhances the sharpness-aware minimization (SAM) technique, utilizing a two-batch strategy to provide a more accurate perturbation estimate and an efficient gradient descent, thereby improving parameter robustness against noise. Notably, DPAdapter can act as a plug-and-play component and be combined with existing DPML algorithms to further improve their performance. Our experiments show that DPAdapter vastly enhances state-of-the-art DPML algorithms, increasing average accuracy from 72.92% to 77.09% with a privacy budget of ε=4.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 56 of its 75 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

56 of 75 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
38 more
  1. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  5. 2021Membership Inference Attacks From First Principlescited by 839
  6. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  7. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  8. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  9. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  10. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  11. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  12. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  13. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  14. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  15. 2020Membership Leakage in Label-Only Exposurescited by 156
  16. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
  17. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  18. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  19. 2022Reconstructing Training Data with Informed Adversariescited by 143
  20. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  21. 2021Deep Learning with Label Differential Privacycited by 113
  22. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  23. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  24. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 79
  25. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  26. 2019Differentially Private Model Publishing for Deep Learningcited by 69
  27. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  28. 2017Efficient Private ERM for Smooth Objectivescited by 61
  29. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  30. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  31. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  32. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 35
  33. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  34. 2022Robust Weight Perturbation for Adversarial Trainingcited by 25
  35. 2021Formalizing Generalization and Robustness of Neural Networks to Weight Perturbationscited by 22
  36. 2023LDPTrace: Locally Differentially Private Trajectory Synthesiscited by 16
  37. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  38. 2020Continuous Release of Data Streams under both Centralized and Local Differential Privacycited by 13

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 1.

7 papers.

  1. 2024SoK: Dataset Copyright Auditing in Machine Learning Systemscited by 14
  2. 2026How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignmentcited by 2
  3. 2024S2NeRF: Privacy-preserving Training Framework for NeRFcited by 1
  4. 2024Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Securitycited by 0
  5. 2024Protecting Confidentiality, Privacy and Integrity in Collaborative Learningcited by 0
  6. 2025PPFL-RDSN: Privacy-Preserving Federated Learning-based Residual Dense Spatial Networks for Encrypted Lossy Image Reconstructioncited by 0
  7. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0