# Towards Understanding the Impact of Model Size on Differential Private Classification

arXiv: https://arxiv.org/abs/2111.13895 · HTML: https://mlsanity.com/paper/2111.13895/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 8 of 8)

- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Adversarially Robust Generalization Requires More Data](https://mlsanity.com/paper/1804.11285/) (2018)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds](https://mlsanity.com/paper/1405.7085/) (2014)
- [More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models](https://mlsanity.com/paper/2002.04725/) (2020)
- [An Efficient DP-SGD Mechanism for Large Scale NLP Models](https://mlsanity.com/paper/2107.14586/) (2021)
- [Sparsifying the Fisher Linear Discriminant by Rotation](https://mlsanity.com/paper/1408.4475/) (2014)

## Built on by (most cited first, 10 of 10)

- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search](https://mlsanity.com/paper/2210.02156/) (2022)
- [PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining](https://mlsanity.com/paper/2311.12850/) (2023)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD](https://mlsanity.com/paper/2601.10237/) (2026)
- [An Optimization Framework for Differentially Private Sparse Fine-Tuning](https://mlsanity.com/paper/2503.12822/) (2025)
- [DP-DCAN: Differentially Private Deep Contrastive Autoencoder Network for Single-cell Clustering](https://mlsanity.com/paper/2311.03410/) (2023)
- [Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients](https://mlsanity.com/paper/2405.17376/) (2024)
- [Differentially Private 2D Human Pose Estimation](https://mlsanity.com/paper/2504.10190/) (2025)
- [PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization](https://mlsanity.com/paper/2605.06505/) (2026)

## Abstract (arXiv)

Differential privacy (DP) is an essential technique for privacy-preserving. It was found that a large model trained for privacy preserving performs worse than a smaller model (e.g. ResNet50 performs worse than ResNet18). To better understand this phenomenon, we study high dimensional DP learning from the viewpoint of generalization. Theoretically, we show that for the simple Gaussian model with even small DP noise, if the dimension is large enough, then the classification error can be as bad as the random guessing. Then we propose a feature selection method to reduce the size of the model, based on a new metric which trades off the classification accuracy and privacy preserving. Experiments on real data support our theoretical results and demonstrate the advantage of the proposed method.