mlsanityData to 7 Oct 2026

DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy

arXiv 2110.08557 ↗First posted 16 Oct 2021cs.LG cs.AI cs.CRcited by 20

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks (DNNs) for meaningful differential privacy (DP) guarantees severely degrades model utility. In this paper, we demonstrate that the architecture of DNNs has a significant impact on model utility in the context of private deep learning, whereas its effect is largely unexplored in previous studies. In light of this missing, we propose the very first framework that employs neural architecture search to automatic model design for private deep learning, dubbed as DPNAS. To integrate private learning with architecture search, we delicately design a novel search space and propose a DP-aware method for training candidate models. We empirically certify the effectiveness of the proposed framework. The searched model DPNASNet achieves state-of-the-art privacy/utility trade-offs, e.g., for the privacy budget of (ε, δ)=(3, 1×10-5), our model obtains test accuracy of 98.57% on MNIST, 88.09% on FashionMNIST, and 68.33% on CIFAR-10. Furthermore, by studying the generated architectures, we provide several intriguing findings of designing private-learning-friendly DNNs, which can shed new light on model design for deep learning with differential privacy.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

37 of 48 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2016Wide Residual Networkscited by 3,057
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
25 more
  1. 2018Group Normalizationcited by 1,514
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  4. 2019Deep Leakage from Gradientscited by 1,004
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2017Large-Scale Evolution of Image Classifierscited by 348
  10. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  11. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  12. 2018Differentially Private Generative Adversarial Networkcited by 235
  13. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  14. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  15. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  16. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  17. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  18. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  19. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  20. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  21. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  22. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  23. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  24. 2018Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budgetcited by 31
  25. 2019P3SGD: Patient Privacy Preserving SGD for Regularizing Deep CNNs in Pathological Image Classificationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 3; 2025: 3; 2026: 2.

20 papers.

Sort:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  2. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  3. 2023DPMLBench: Holistic Evaluation of Differentially Private Machine Learningcited by 13
  4. 2023An Empirical Analysis of Fairness Notions under Differential Privacycited by 9
  5. 2023Differentially Private Video Activity Recognitioncited by 9
  6. 2023Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filteringcited by 6
  7. 2023Differential Privacy Meets Neural Network Pruningcited by 5
  8. 2022On the Importance of Architecture and Feature Selection in Differentially Private Machine Learningcited by 3
  9. 2023Divide-and-Conquer the NAS puzzle in Resource Constrained Federated Learning Systemscited by 3
  10. 2023Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classificationcited by 3
  11. 2024Wasserstein Differential Privacycited by 3
  12. 2025SoK: What Makes Private Learning Unfair?cited by 3
8 more
  1. 2022CR-LSO: Convex Neural Architecture Optimization in the Latent Space of Graph Variational Autoencoder with Input Convex Neural Networkscited by 2
  2. 2025Global optimization of graph acquisition functions for neural architecture searchcited by 2
  3. 2022SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealingcited by 1
  4. 2024Pushing Joint Image Denoising and Classification to the Edgecited by 0
  5. 2024Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Securitycited by 0
  6. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  7. 2026Privacy-Preserving Federated Learning via Differential Privacy and Homomorphic Encryption for Cardiovascular Disease Risk Modelingcited by 0
  8. 2026End-to-End Differential Privacy in Training Deep Neural Network Classifierscited by 0