# Regularising for invariance to data augmentation improves supervised learning

arXiv: https://arxiv.org/abs/2203.03304 · HTML: https://mlsanity.com/paper/2203.03304/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Representation Learning via Invariant Causal Mechanisms](https://mlsanity.com/paper/2010.07922/) (2020)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [On the Generalization Benefit of Noise in Stochastic Gradient Descent](https://mlsanity.com/paper/2006.15081/) (2020)
- [Learning Loss for Test-Time Augmentation](https://mlsanity.com/paper/2010.11422/) (2020)
- [Learning Invariances using the Marginal Likelihood](https://mlsanity.com/paper/1808.05563/) (2018)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [Sample Efficiency of Data Augmentation Consistency Regularization](https://mlsanity.com/paper/2202.12230/) (2022)

## Built on by (most cited first, 9 of 9)

- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [SequenceMatch: Revisiting the design of weak-strong augmentations for Semi-supervised learning](https://mlsanity.com/paper/2310.15787/) (2023)
- [Generating Physical Dynamics under Priors](https://mlsanity.com/paper/2409.00730/) (2024)
- [Image augmentation with conformal mappings for a convolutional neural network](https://mlsanity.com/paper/2212.05258/) (2022)
- [Subspace-Configurable Networks](https://mlsanity.com/paper/2305.13536/) (2023)
- [Contextual Reliability: When Different Features Matter in Different Contexts](https://mlsanity.com/paper/2307.10026/) (2023)
- [From One to the Power of Many: Invariance to Multi-LiDAR Perception from Single-Sensor Datasets](https://mlsanity.com/paper/2409.18592/) (2024)
- [The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalization](https://mlsanity.com/paper/2602.14423/) (2026)

## Abstract (arXiv)

Data augmentation is used in machine learning to make the classifier invariant to label-preserving transformations. Usually this invariance is only encouraged implicitly by including a single augmented input during training. However, several works have recently shown that using multiple augmentations per input can improve generalisation or can be used to incorporate invariances more explicitly. In this work, we first empirically compare these recently proposed objectives that differ in whether they rely on explicit or implicit regularisation and at what level of the predictor they encode the invariances. We show that the predictions of the best performing method are also the most similar when compared on different augmentations of the same input. Inspired by this observation, we propose an explicit regulariser that encourages this invariance on the level of individual model predictions. Through extensive experiments on CIFAR-100 and ImageNet we show that this explicit regulariser (i) improves generalisation and (ii) equalises performance differences between all considered objectives. Our results suggest that objectives that encourage invariance on the level of the neural network itself generalise better than those that achieve invariance by averaging predictions of non-invariant models.