# Grounding inductive biases in natural images:invariance stems from variations in data

arXiv: https://arxiv.org/abs/2106.05121 · HTML: https://mlsanity.com/paper/2106.05121/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data](https://mlsanity.com/paper/2002.12880/) (2020)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Meta-Learning Symmetries by Reparameterization](https://mlsanity.com/paper/2007.02933/) (2020)
- [Learning Invariances in Neural Networks](https://mlsanity.com/paper/2010.11882/) (2020)
- [Learning Invariances using the Marginal Likelihood](https://mlsanity.com/paper/1808.05563/) (2018)
- [Quantifying Translation-Invariance in Convolutional Neural Networks](https://mlsanity.com/paper/1801.01450/) (2017)
- [Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentation](https://mlsanity.com/paper/1510.02795/) (2015)
- [RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks](https://mlsanity.com/paper/1805.06846/) (2018)
- [Robustness to Augmentations as a Generalization metric](https://mlsanity.com/paper/2101.06459/) (2021)
- [Unsupervised Transformation Learning via Convex Relaxations](https://mlsanity.com/paper/1711.02226/) (2017)
- [Tracking translation invariance in CNNs](https://mlsanity.com/paper/2104.05997/) (2021)

## Built on by (most cited first, 27 of 27)

- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [The Lie Derivative for Measuring Learned Equivariance](https://mlsanity.com/paper/2210.02984/) (2022)
- [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022)
- [3D molecule generation by denoising voxel grids](https://mlsanity.com/paper/2306.07473/) (2023)
- [PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning](https://mlsanity.com/paper/2308.03977/) (2023)
- [The Robustness Limits of SoTA Vision Models to Natural Variation](https://mlsanity.com/paper/2210.13604/) (2022)
- [What Affects Learned Equivariance in Deep Image Recognition Models?](https://mlsanity.com/paper/2304.02628/) (2023)
- [Equivariance and Invariance Inductive Bias for Learning from Insufficient Data](https://mlsanity.com/paper/2207.12258/) (2022)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [TABASCO: A Fast, Simplified Model for Molecular Generation with Improved Physical Quality](https://mlsanity.com/paper/2507.00899/) (2025)
- [When Representations Align: Universality in Representation Learning Dynamics](https://mlsanity.com/paper/2402.09142/) (2024)
- [A Generative Model of Symmetry Transformations](https://mlsanity.com/paper/2403.01946/) (2024)
- [Regularising for invariance to data augmentation improves supervised learning](https://mlsanity.com/paper/2203.03304/) (2022)
- [Deep invariant networks with differentiable augmentation layers](https://mlsanity.com/paper/2202.02142/) (2022)
- [Test-Time Canonicalization by Foundation Models for Robust Perception](https://mlsanity.com/paper/2507.10375/) (2025)
- [Robust Self-Supervised Learning with Lie Groups](https://mlsanity.com/paper/2210.13356/) (2022)
- [A Principled Framework for Multi-View Contrastive Learning](https://mlsanity.com/paper/2507.06979/) (2025)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Learning to Transform for Generalizable Instance-wise Invariance](https://mlsanity.com/paper/2309.16672/) (2023)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [Sparse Data Augmentation for Optimization with Provable Guarantees](https://mlsanity.com/paper/2609.08133/) (2026)
- [Symmetry Defense Against CNN Adversarial Perturbation Attacks](https://mlsanity.com/paper/2210.04087/) (2022)
- [Symmetry Defense Against XGBoost Adversarial Perturbation Attacks](https://mlsanity.com/paper/2308.05575/) (2023)
- [From One to the Power of Many: Invariance to Multi-LiDAR Perception from Single-Sensor Datasets](https://mlsanity.com/paper/2409.18592/) (2024)
- [Data Distributional Properties As Inductive Bias for Systematic Generalization](https://mlsanity.com/paper/2502.20499/) (2025)
- [You May Be Running the Wrong Inception Crop](https://mlsanity.com/paper/2610.04147/) (2026)
- [Measuring Learned Equivariance: An Information Theoretic Approach](https://mlsanity.com/paper/2610.17269/) (2026)

## Abstract (arXiv)

To perform well on unseen and potentially out-of-distribution samples, it is desirable for machine learning models to have a predictable response with respect to transformations affecting the factors of variation of the input. Here, we study the relative importance of several types of inductive biases towards such predictable behavior: the choice of data, their augmentations, and model architectures. Invariance is commonly achieved through hand-engineered data augmentation, but do standard data augmentations address transformations that explain variations in real data? While prior work has focused on synthetic data, we attempt here to characterize the factors of variation in a real dataset, ImageNet, and study the invariance of both standard residual networks and the recently proposed vision transformer with respect to changes in these factors. We show standard augmentation relies on a precise combination of translation and scale, with translation recapturing most of the performance improvement -- despite the (approximate) translation invariance built in to convolutional architectures, such as residual networks. In fact, we found that scale and translation invariance was similar across residual networks and vision transformer models despite their markedly different architectural inductive biases. We show the training data itself is the main source of invariance, and that data augmentation only further increases the learned invariances. Notably, the invariances learned during training align with the ImageNet factors of variation we found. Finally, we find that the main factors of variation in ImageNet mostly relate to appearance and are specific to each class.