mlsanityData to 7 Oct 2026

In What Ways Are Deep Neural Networks Invariant and How Should We Measure This?

arXiv 2210.03773 ↗First posted 7 Oct 2022cs.LG cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

It is often said that a deep learning model is "invariant" to some specific type of transformation. However, what is meant by this statement strongly depends on the context in which it is made. In this paper we explore the nature of invariance and equivariance of deep learning models with the goal of better understanding the ways in which they actually capture these concepts on a formal level. We introduce a family of invariance and equivariance metrics that allows us to quantify these properties in a way that disentangles them from other metrics such as loss or accuracy. We use our metrics to better understand the two most popular methods used to build invariance into networks: data augmentation and equivariant layers. We draw a range of conclusions about invariance and equivariance in deep learning models, ranging from whether initializing a model with pretrained weights has an effect on a trained model's invariance, to the extent to which invariance learned via training can generalize to out-of-distribution data.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

35 of 61 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2016Group Equivariant Convolutional Networkscited by 1,154
  7. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  10. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  11. 2018Spherical CNNscited by 452
  12. 2019General E(2)-Equivariant Steerable CNNscited by 441
23 more
  1. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  2. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  3. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  4. 2016Steerable CNNscited by 298
  5. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  6. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  7. 2017Data Augmentation Generative Adversarial Networkscited by 278
  8. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244
  9. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 193
  10. 2018xView: Objects in Context in Overhead Imagerycited by 145
  11. 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
  12. 2019Scale-Equivariant Steerable Networkscited by 110
  13. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  14. 2016Rotation equivariant vector field networkscited by 98
  15. 2017Learning to Compose Domain-Specific Transformations for Data Augmentationcited by 91
  16. 2020On the Benefits of Invariance in Neural Networkscited by 78
  17. 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
  18. 2020Efficient Generalized Spherical CNNscited by 29
  19. 2022iDECODe: In-distribution Equivariance for Conformal Out-of-distribution Detectioncited by 27
  20. 2015Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentationcited by 24
  21. 2016RenderGAN: Generating Realistic Labeled Datacited by 20
  22. 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
  23. 2019A Group-Theoretic Framework for Data Augmentationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3; 2026: 2.

9 papers.

  1. 2023On genuine invariance learning without weight-tyingcited by 13
  2. 2024Relaxed Equivariance via Multitask Learningcited by 13
  3. 2024Understanding the Role of Invariance in Transfer Learningcited by 1
  4. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1
  5. 2025Robust Canonicalization through Bootstrapped Data Re-Alignmentcited by 1
  6. 2025Training Dynamics of Learning 3D-Rotational Equivariancecited by 1
  7. 2022From CNNs to Shift-Invariant Twin Models Based on Complex Waveletscited by 0
  8. 2026SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representationscited by 0
  9. 2026Sparse Data Augmentation for Optimization with Provable Guaranteescited by 0