# DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy

arXiv: https://arxiv.org/abs/2110.08557 · HTML: https://mlsanity.com/paper/2110.08557/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Subsampled R'enyi Differential Privacy and Analytical Moments Accountant](https://mlsanity.com/paper/1808.00087/) (2018)
- [Differentially Private Generative Adversarial Network](https://mlsanity.com/paper/1802.06739/) (2018)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [R'enyi Differential Privacy of the Sampled Gaussian Mechanism](https://mlsanity.com/paper/1908.10530/) (2019)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)

## Built on by (most cited first, 20 of 20)

- [How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy](https://mlsanity.com/paper/2303.00654/) (2023)
- [Differentially Private Sharpness-Aware Training](https://mlsanity.com/paper/2306.05651/) (2023)
- [DPMLBench: Holistic Evaluation of Differentially Private Machine Learning](https://mlsanity.com/paper/2305.05900/) (2023)
- [An Empirical Analysis of Fairness Notions under Differential Privacy](https://mlsanity.com/paper/2302.02910/) (2023)
- [Differentially Private Video Activity Recognition](https://mlsanity.com/paper/2306.15742/) (2023)
- [Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filtering](https://mlsanity.com/paper/2307.13231/) (2023)
- [Differential Privacy Meets Neural Network Pruning](https://mlsanity.com/paper/2303.04612/) (2023)
- [On the Importance of Architecture and Feature Selection in Differentially Private Machine Learning](https://mlsanity.com/paper/2205.06720/) (2022)
- [Divide-and-Conquer the NAS puzzle in Resource Constrained Federated Learning Systems](https://mlsanity.com/paper/2305.07135/) (2023)
- [Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classification](https://mlsanity.com/paper/2306.11754/) (2023)
- [Wasserstein Differential Privacy](https://mlsanity.com/paper/2401.12436/) (2024)
- [SoK: What Makes Private Learning Unfair?](https://mlsanity.com/paper/2501.14414/) (2025)
- [CR-LSO: Convex Neural Architecture Optimization in the Latent Space of Graph Variational Autoencoder with Input Convex Neural Networks](https://mlsanity.com/paper/2211.05950/) (2022)
- [Global optimization of graph acquisition functions for neural architecture search](https://mlsanity.com/paper/2505.23640/) (2025)
- [SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealing](https://mlsanity.com/paper/2211.07218/) (2022)
- [Pushing Joint Image Denoising and Classification to the Edge](https://mlsanity.com/paper/2409.08943/) (2024)
- [Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Security](https://mlsanity.com/paper/2409.11663/) (2024)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [Privacy-Preserving Federated Learning via Differential Privacy and Homomorphic Encryption for Cardiovascular Disease Risk Modeling](https://mlsanity.com/paper/2604.27598/) (2026)
- [End-to-End Differential Privacy in Training Deep Neural Network Classifiers](https://mlsanity.com/paper/2607.19580/) (2026)

## Abstract (arXiv)

Training deep neural networks (DNNs) for meaningful differential privacy (DP) guarantees severely degrades model utility. In this paper, we demonstrate that the architecture of DNNs has a significant impact on model utility in the context of private deep learning, whereas its effect is largely unexplored in previous studies. In light of this missing, we propose the very first framework that employs neural architecture search to automatic model design for private deep learning, dubbed as DPNAS. To integrate private learning with architecture search, we delicately design a novel search space and propose a DP-aware method for training candidate models. We empirically certify the effectiveness of the proposed framework. The searched model DPNASNet achieves state-of-the-art privacy/utility trade-offs, e.g., for the privacy budget of (ε, δ)=(3, 1×10-5), our model obtains test accuracy of 98.57% on MNIST, 88.09% on FashionMNIST, and 68.33% on CIFAR-10. Furthermore, by studying the generated architectures, we provide several intriguing findings of designing private-learning-friendly DNNs, which can shed new light on model design for deep learning with differential privacy.