# Why do deep convolutional networks generalize so poorly to small image transformations?

arXiv: https://arxiv.org/abs/1805.12177 · HTML: https://mlsanity.com/paper/1805.12177/

## Key facts

- Cited by 198 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 30 of 198)

- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation](https://mlsanity.com/paper/2106.04732/) (2021)
- [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025)
- [AugMax: Adversarial Composition of Random Augmentations for Robust Training](https://mlsanity.com/paper/2110.13771/) (2021)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Equivariant Point Network for 3D Point Cloud Analysis](https://mlsanity.com/paper/2103.14147/) (2021)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning](https://mlsanity.com/paper/2106.15831/) (2021)
- [On the effectiveness of adversarial training against common corruptions](https://mlsanity.com/paper/2103.02325/) (2021)
- [Probable Domain Generalization via Quantile Risk Minimization](https://mlsanity.com/paper/2207.09944/) (2022)
- [MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption](https://mlsanity.com/paper/2103.16201/) (2021)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [The Lie Derivative for Measuring Learned Equivariance](https://mlsanity.com/paper/2210.02984/) (2022)
- [Ensembling with Deep Generative Views](https://mlsanity.com/paper/2104.14551/) (2021)
- [Assaying Out-Of-Distribution Generalization in Transfer Learning](https://mlsanity.com/paper/2207.09239/) (2022)
- [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://mlsanity.com/paper/2104.11222/) (2021)
- [On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence](https://mlsanity.com/paper/2207.04630/) (2022)
- [Shift-tolerant Perceptual Similarity Metric](https://mlsanity.com/paper/2207.13686/) (2022)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [WaveCNet: Wavelet Integrated CNNs to Suppress Aliasing Effect for Noise-Robust Image Classification](https://mlsanity.com/paper/2107.13335/) (2021)
- [Grounding inductive biases in natural images:invariance stems from variations in data](https://mlsanity.com/paper/2106.05121/) (2021)
- [Dense Unsupervised Learning for Video Segmentation](https://mlsanity.com/paper/2111.06265/) (2021)
- [Benchmarking the Robustness of Deep Neural Networks to Common Corruptions in Digital Pathology](https://mlsanity.com/paper/2206.14973/) (2022)
- [Impact of Aliasing on Generalization in Deep Convolutional Networks](https://mlsanity.com/paper/2108.03489/) (2021)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) are commonly assumed to be invariant to small image transformations: either because of the convolutional architecture or because they were trained using data augmentation. Recently, several authors have shown that this is not the case: small translations or rescalings of the input image can drastically change the network's prediction. In this paper, we quantify this phenomena and ask why neither the convolutional architecture nor data augmentation are sufficient to achieve the desired invariance. Specifically, we show that the convolutional architecture does not give invariance since architectures ignore the classical sampling theorem, and data augmentation does not give invariance because the CNNs learn to be invariant to transformations only for images that are very similar to typical images from the training set. We discuss two possible solutions to this problem: (1) antialiasing the intermediate representations and (2) increasing data augmentation and show that they provide only a partial solution at best. Taken together, our results indicate that the problem of insuring invariance to small image transformations in neural networks while preserving high accuracy remains unsolved.