# Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error

arXiv: https://arxiv.org/abs/2105.13343 · HTML: https://mlsanity.com/paper/2105.13343/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 38)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning](https://mlsanity.com/paper/1712.06559/) (2017)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model](https://mlsanity.com/paper/1907.04164/) (2019)

## Built on by (most cited first, 30 of 33)

- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [VOLO: Vision Outlooker for Visual Recognition](https://mlsanity.com/paper/2106.13112/) (2021)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Differentially private training of residual networks with scale normalisation](https://mlsanity.com/paper/2203.00324/) (2022)
- [Extreme Masking for Learning Instance and Distributed Visual Representations](https://mlsanity.com/paper/2206.04667/) (2022)
- [Using Soft Labels to Model Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2109.12622/) (2021)
- [Deep AutoAugment](https://mlsanity.com/paper/2203.06172/) (2022)
- [Optimizers Qualitatively Alter Solutions And We Should Leverage This](https://mlsanity.com/paper/2507.12224/) (2025)
- [Unlocking Accuracy and Fairness in Differentially Private Image Classification](https://mlsanity.com/paper/2308.10888/) (2023)
- [ConvNets Match Vision Transformers at Scale](https://mlsanity.com/paper/2310.16764/) (2023)
- [Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness](https://mlsanity.com/paper/2408.05446/) (2024)
- [Using Diffusion Models to Generate Synthetic Labelled Data for Medical Image Segmentation](https://mlsanity.com/paper/2310.16794/) (2023)
- [Regularising for invariance to data augmentation improves supervised learning](https://mlsanity.com/paper/2203.03304/) (2022)
- [Poly-View Contrastive Learning](https://mlsanity.com/paper/2403.05490/) (2024)
- [Disjoint Masking with Joint Distillation for Efficient Masked Image Modeling](https://mlsanity.com/paper/2301.00230/) (2022)
- [Tied-Augment: Controlling Representation Similarity Improves Data Augmentation](https://mlsanity.com/paper/2305.13520/) (2023)
- [Encoder-Decoder Architectures for Clinically Relevant Coronary Artery Segmentation](https://mlsanity.com/paper/2106.11447/) (2021)
- [Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models](https://mlsanity.com/paper/2301.13104/) (2023)
- [A Unified View of Differentially Private Deep Generative Modeling](https://mlsanity.com/paper/2309.15696/) (2023)
- [SpliceMix: A Cross-scale and Semantic Blending Augmentation Strategy for Multi-label Image Classification](https://mlsanity.com/paper/2311.15200/) (2023)
- [A Principled Framework for Multi-View Contrastive Learning](https://mlsanity.com/paper/2507.06979/) (2025)
- [SmoothNets: Optimizing CNN architecture design for differentially private deep learning](https://mlsanity.com/paper/2205.04095/) (2022)
- [Multi-attacks: Many images + the same adversarial attack → many target labels](https://mlsanity.com/paper/2308.03792/) (2023)
- [Aligned Contrastive Loss for Long-Tailed Recognition](https://mlsanity.com/paper/2506.01071/) (2025)
- [Adults as Augmentations for Children in Facial Emotion Recognition with Contrastive Learning](https://mlsanity.com/paper/2202.05187/) (2022)

## Abstract (arXiv)

In computer vision, it is standard practice to draw a single sample from the data augmentation procedure for each unique image in the mini-batch. However recent work has suggested drawing multiple samples can achieve higher test accuracies. In this work, we provide a detailed empirical evaluation of how the number of augmentation samples per unique image influences model performance on held out data when training deep ResNets. We demonstrate drawing multiple samples per image consistently enhances the test accuracy achieved for both small and large batch training. Crucially, this benefit arises even if different numbers of augmentations per image perform the same number of parameter updates and gradient evaluations (requiring the same total compute). Although prior work has found variance in the gradient estimate arising from subsampling the dataset has an implicit regularization benefit, our experiments suggest variance which arises from the data augmentation process harms generalization. We apply these insights to the highly performant NFNet-F5, achieving 86.8% top-1 w/o extra data on ImageNet.