# Equivariance and Invariance Inductive Bias for Learning from Insufficient Data

arXiv: https://arxiv.org/abs/2207.12258 · HTML: https://mlsanity.com/paper/2207.12258/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 93 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 30 of 70)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)

## Built on by (most cited first, 14 of 14)

- [Neuron Structure Modeling for Generalizable Remote Physiological Measurement](https://mlsanity.com/paper/2303.05955/) (2023)
- [Identifying Hard Noise in Long-Tailed Sample Distribution](https://mlsanity.com/paper/2207.13378/) (2022)
- [Siamese Vision Transformers are Scalable Audio-visual Learners](https://mlsanity.com/paper/2403.19638/) (2024)
- [Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2309.12742/) (2023)
- [Self-Supervised Learning for Place Representation Generalization across Appearance Changes](https://mlsanity.com/paper/2303.02370/) (2023)
- [Disentangling Masked Autoencoders for Unsupervised Domain Generalization](https://mlsanity.com/paper/2407.07544/) (2024)
- [Generalized Spherical Neural Operators: Green's Function Formulation](https://mlsanity.com/paper/2512.10723/) (2025)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Local Lesion Generation is Effective for Capsule Endoscopy Image Data Augmentation in a Limited Data Setting](https://mlsanity.com/paper/2411.03098/) (2024)
- [Learning Intra and Inter-Camera Invariance for Isolated Camera Supervised Person Re-identification](https://mlsanity.com/paper/2311.01155/) (2023)
- [Invariant Correlation of Representation with Label: Enhancing Domain Generalization in Noisy Environments](https://mlsanity.com/paper/2407.01749/) (2024)
- [Breaker: Removing Shortcut Cues with User Clustering for Single-slot Recommendation System](https://mlsanity.com/paper/2506.00828/) (2025)
- [Modality-Aware Bias Mitigation and Invariance Learning for Unsupervised Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2512.07760/) (2025)
- [Factorized Neural Operators Decompose Dynamic and Persistent Responses](https://mlsanity.com/paper/2606.16900/) (2026)

## Abstract (arXiv)

We are interested in learning robust models from insufficient data, without the need for any externally pre-trained checkpoints. First, compared to sufficient data, we show why insufficient data renders the model more easily biased to the limited training environments that are usually different from testing. For example, if all the training swan samples are "white", the model may wrongly use the "white" environment to represent the intrinsic class swan. Then, we justify that equivariance inductive bias can retain the class feature while invariance inductive bias can remove the environmental feature, leaving the class feature that generalizes to any environmental changes in testing. To impose them on learning, for equivariance, we demonstrate that any off-the-shelf contrastive-based self-supervised feature learning method can be deployed; for invariance, we propose a class-wise invariant risk minimization (IRM) that efficiently tackles the challenge of missing environmental annotation in conventional IRM. State-of-the-art experimental results on real-world benchmarks (VIPriors, ImageNet100 and NICO) validate the great potential of equivariance and invariance in data-efficient learning. The code is available at https://github.com/Wangt-CN/EqInv