# A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments

arXiv: https://arxiv.org/abs/2202.08325 · HTML: https://mlsanity.com/paper/2202.08325/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Implicit Regularization in Deep Learning May Not Be Explainable by Norms](https://mlsanity.com/paper/2005.06398/) (2020)
- [Three Mechanisms of Weight Decay Regularization](https://mlsanity.com/paper/1810.12281/) (2018)
- [Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1709.01953/) (2017)
- [The Implicit and Explicit Regularization Effects of Dropout](https://mlsanity.com/paper/2002.12915/) (2020)
- [Data augmentation instead of explicit regularization](https://mlsanity.com/paper/1806.03852/) (2018)
- [A representer theorem for deep neural networks](https://mlsanity.com/paper/1802.09210/) (2018)
- [Dropout as data augmentation](https://mlsanity.com/paper/1506.08700/) (2015)
- [Further advantages of data augmentation on convolutional neural networks](https://mlsanity.com/paper/1906.11052/) (2019)
- [Do deep nets really need weight decay and dropout?](https://mlsanity.com/paper/1802.07042/) (2018)

## Built on by (most cited first, 14 of 14)

- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries](https://mlsanity.com/paper/2302.12828/) (2023)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Is Self-Supervised Learning More Robust Than Supervised Learning?](https://mlsanity.com/paper/2206.05259/) (2022)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [Poly-View Contrastive Learning](https://mlsanity.com/paper/2403.05490/) (2024)
- [A Survey of Methods, Challenges and Perspectives in Causality](https://mlsanity.com/paper/2302.00293/) (2023)
- [Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space](https://mlsanity.com/paper/2311.15022/) (2023)
- [SeqAug: Sequential Feature Resampling as a modality agnostic augmentation method](https://mlsanity.com/paper/2305.01954/) (2023)
- [SpeckleNN: A unified embedding for real-time speckle pattern classification in X-ray single-particle imaging with limited labeled examples](https://mlsanity.com/paper/2302.06895/) (2023)
- [Understanding the Role of Invariance in Transfer Learning](https://mlsanity.com/paper/2407.04325/) (2024)
- [Trustworthiness of Laser-Induced Breakdown Spectroscopy Predictions via Simulation-based Synthetic Data Augmentation and Multitask Learning](https://mlsanity.com/paper/2210.03762/) (2022)

## Abstract (arXiv)

Data-Augmentation (DA) is known to improve performance across tasks and datasets. We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA? How does the augmentation policy impact the final parameters of a model? We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model's output under a given DA distribution. Those derivations open new avenues to quantify the benefits and limitations of DA. For example, we show that common DAs require tens of thousands of samples for the loss at hand to be correctly estimated and for the model training to converge. We show that for a training loss to be stable under DA sampling, the model's saliency map (gradient of the loss with respect to the model's input) must align with the smallest eigenvector of the sample variance under the considered DA augmentation, hinting at a possible explanation on why models tend to shift their focus from edges to textures.