# In What Ways Are Deep Neural Networks Invariant and How Should We Measure This?

arXiv: https://arxiv.org/abs/2210.03773 · HTML: https://mlsanity.com/paper/2210.03773/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Learning Steerable Filters for Rotation Equivariant CNNs](https://mlsanity.com/paper/1711.07289/) (2017)
- [xView: Objects in Context in Overhead Imagery](https://mlsanity.com/paper/1802.07856/) (2018)
- [Deep Scale-spaces: Equivariance Over Scale](https://mlsanity.com/paper/1905.11697/) (2019)
- [Scale-Equivariant Steerable Networks](https://mlsanity.com/paper/1910.11093/) (2019)
- [LieTransformer: Equivariant self-attention for Lie Groups](https://mlsanity.com/paper/2012.10885/) (2020)
- [Rotation equivariant vector field networks](https://mlsanity.com/paper/1612.09346/) (2016)
- [Learning to Compose Domain-Specific Transformations for Data Augmentation](https://mlsanity.com/paper/1709.01643/) (2017)
- [On the Benefits of Invariance in Neural Networks](https://mlsanity.com/paper/2005.00178/) (2020)
- [Excessive Invariance Causes Adversarial Vulnerability](https://mlsanity.com/paper/1811.00401/) (2018)
- [Efficient Generalized Spherical CNNs](https://mlsanity.com/paper/2010.11661/) (2020)

## Built on by (most cited first, 9 of 9)

- [On genuine invariance learning without weight-tying](https://mlsanity.com/paper/2308.03904/) (2023)
- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [Understanding the Role of Invariance in Transfer Learning](https://mlsanity.com/paper/2407.04325/) (2024)
- [How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2505.16581/) (2025)
- [Robust Canonicalization through Bootstrapped Data Re-Alignment](https://mlsanity.com/paper/2510.08178/) (2025)
- [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025)
- [SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations](https://mlsanity.com/paper/2602.04054/) (2026)
- [Sparse Data Augmentation for Optimization with Provable Guarantees](https://mlsanity.com/paper/2609.08133/) (2026)
- [From CNNs to Shift-Invariant Twin Models Based on Complex Wavelets](https://mlsanity.com/paper/2212.00394/) (2022)

## Abstract (arXiv)

It is often said that a deep learning model is "invariant" to some specific type of transformation. However, what is meant by this statement strongly depends on the context in which it is made. In this paper we explore the nature of invariance and equivariance of deep learning models with the goal of better understanding the ways in which they actually capture these concepts on a formal level. We introduce a family of invariance and equivariance metrics that allows us to quantify these properties in a way that disentangles them from other metrics such as loss or accuracy. We use our metrics to better understand the two most popular methods used to build invariance into networks: data augmentation and equivariant layers. We draw a range of conclusions about invariance and equivariance in deep learning models, ranging from whether initializing a model with pretrained weights has an effect on a trained model's invariance, to the extent to which invariance learned via training can generalize to out-of-distribution data.