# Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design

arXiv: https://arxiv.org/abs/2111.14924 · HTML: https://mlsanity.com/paper/2111.14924/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [Local Differential Privacy for Deep Learning](https://mlsanity.com/paper/1908.02997/) (2019)
- [Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning](https://mlsanity.com/paper/1709.05750/) (2017)
- [Three Tools for Practical Differential Privacy](https://mlsanity.com/paper/1812.02890/) (2018)

## Built on by (most cited first, 6 of 6)

- [Differentially private training of residual networks with scale normalisation](https://mlsanity.com/paper/2203.00324/) (2022)
- [Differentially Private Video Activity Recognition](https://mlsanity.com/paper/2306.15742/) (2023)
- [SmoothNets: Optimizing CNN architecture design for differentially private deep learning](https://mlsanity.com/paper/2205.04095/) (2022)
- [On the Importance of Architecture and Feature Selection in Differentially Private Machine Learning](https://mlsanity.com/paper/2205.06720/) (2022)
- [Automated discovery of trade-off between utility, privacy and fairness in machine learning models](https://mlsanity.com/paper/2311.15691/) (2023)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)

## Abstract (arXiv)

One barrier to more widespread adoption of differentially private neural networks is the entailed accuracy loss. To address this issue, the relationship between neural network architectures and model accuracy under differential privacy constraints needs to be better understood. As a first step, we test whether extant knowledge on architecture design also holds in the differentially private setting. Our findings show that it does not; architectures that perform well without differential privacy, do not necessarily do so with differential privacy. Consequently, extant knowledge on neural network architecture design cannot be seamlessly translated into the differential privacy context. Future research is required to better understand the relationship between neural network architectures and model accuracy to enable better architecture design choices under differential privacy constraints.