# Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

arXiv: https://arxiv.org/abs/1603.05691 · HTML: https://mlsanity.com/paper/1603.05691/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2016.

## Built on by (most cited first, 8 of 8)

- [Efficient Medical Image Segmentation Based on Knowledge Distillation](https://mlsanity.com/paper/2108.09987/) (2021)
- [Self-Regulation for Semantic Segmentation](https://mlsanity.com/paper/2108.09702/) (2021)
- [Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal](https://mlsanity.com/paper/2306.09008/) (2023)
- [Frugal Machine Learning](https://mlsanity.com/paper/2111.03731/) (2021)
- [MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets](https://mlsanity.com/paper/2108.02559/) (2021)
- [Label Augmentation via Time-based Knowledge Distillation for Financial Anomaly Detection](https://mlsanity.com/paper/2101.01689/) (2021)
- [Federated Semi-Supervised Domain Adaptation via Knowledge Transfer](https://mlsanity.com/paper/2207.10727/) (2022)
- [Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications](https://mlsanity.com/paper/2312.16799/) (2023)

## Abstract (arXiv)

Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or shallow models of high accuracy to be trained. Although previous research showed that shallow feed-forward nets sometimes can learn the complex functions previously learned by deep nets while using the same number of parameters as the deep models they mimic, in this paper we demonstrate that the same methods cannot be used to train accurate models on CIFAR-10 unless the student models contain multiple layers of convolution. Although the student models do not have to be as deep as the teacher model they mimic, the students need multiple convolutional layers to learn functions of comparable accuracy as the deep convolutional teacher.