# Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNs

arXiv: https://arxiv.org/abs/1804.04438 · HTML: https://mlsanity.com/paper/1804.04438/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2018.

## Built on by (most cited first, 8 of 8)

- [Learning sparse features can lead to overfitting in neural networks](https://mlsanity.com/paper/2206.12314/) (2022)
- [The Robustness Limits of SoTA Vision Models to Natural Variation](https://mlsanity.com/paper/2210.13604/) (2022)
- [What Affects Learned Equivariance in Deep Image Recognition Models?](https://mlsanity.com/paper/2304.02628/) (2023)
- [Relative stability toward diffeomorphisms indicates performance in deep nets](https://mlsanity.com/paper/2105.02468/) (2021)
- [How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model](https://mlsanity.com/paper/2404.10727/) (2024)
- [How deep convolutional neural networks lose spatial information with training](https://mlsanity.com/paper/2210.01506/) (2022)
- [Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities](https://mlsanity.com/paper/2409.13154/) (2024)
- [A Structural Approach to the Design of Domain Specific Neural Network Architectures](https://mlsanity.com/paper/2301.09381/) (2023)

## Abstract (arXiv)

Many of our core assumptions about how neural networks operate remain empirically untested. One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks. For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers. Recently, however, interleaved pooling has largely been abandoned. This raises a number of questions: Are our intuitions about deformation stability right at all? Is it important? Is pooling necessary for deformation invariance? If not, how is deformation invariance achieved in its absence? In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers. (2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters. (3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification. (4) Pooling confers too much deformation stability for image classification at initialization, and during training, networks have to learn to counteract this inductive bias. Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.