# SmoothNets: Optimizing CNN architecture design for differentially private deep learning

arXiv: https://arxiv.org/abs/2205.04095 · HTML: https://mlsanity.com/paper/2205.04095/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Augment your batch: better training with larger batches](https://mlsanity.com/paper/1901.09335/) (2019)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [Differentially private training of residual networks with scale normalisation](https://mlsanity.com/paper/2203.00324/) (2022)
- [Not all noise is accounted equally: How differentially private learning benefits from large sampling rates](https://mlsanity.com/paper/2110.06255/) (2021)
- [Differentially private federated deep learning for multi-site medical image segmentation](https://mlsanity.com/paper/2107.02586/) (2021)
- [Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design](https://mlsanity.com/paper/2111.14924/) (2021)

## Built on by (most cited first, 3 of 3)

- [R+R:Understanding Hyperparameter Effects in DP-SGD](https://mlsanity.com/paper/2411.02051/) (2024)
- [Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2210.00053/) (2022)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)

## Abstract (arXiv)

The arguably most widely employed algorithm to train deep neural networks with Differential Privacy is DPSGD, which requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. Empirically, it can be observed that this accuracy degradation is strongly dependent on the model architecture. We investigated this phenomenon and, by combining components which exhibit good individual performance, distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training. Experimentally, we benchmark SmoothNet against standard architectures on two benchmark datasets and observe that our architecture outperforms others, reaching an accuracy of 73.5% on CIFAR-10 at varepsilon=7.0 and 69.2% at varepsilon=7.0 on ImageNette, a state-of-the-art result compared to prior architectural modifications for DP.