mlsanityData to 7 Oct 2026

How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization

arXiv 2210.06441 ↗First posted 12 Oct 2022cs.LG cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

Despite the clear performance benefits of data augmentations, little is known about why they are so effective. In this paper, we disentangle several key mechanisms through which data augmentations operate. Establishing an exchange rate between augmented and additional real data, we find that in out-of-distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data. Moreover, we find that data augmentations which encourage invariances can be more valuable than invariance alone, especially on small and medium sized training sets. Following this observation, we show that augmentations induce additional stochasticity during training, effectively flattening the loss landscape.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

45 of 65 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  12. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
33 more
  1. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  2. 2019General E(2)-Equivariant Steerable CNNscited by 441
  3. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  4. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  5. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
  6. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  7. 2017Three Factors Influencing Minima in SGDcited by 270
  8. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  9. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  10. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  11. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  12. 2022Patches Are All You Need?cited by 201
  13. 2016Deep Pyramidal Residual Networkscited by 163
  14. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  15. 2022A Comprehensive Survey of Image Augmentation Techniques for Deep Learningcited by 95
  16. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  17. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  18. 2019Understanding Generalization through Visualizationscited by 54
  19. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  20. 2020Affinity and Diversity: Quantifying Mechanisms of Data Augmentationcited by 45
  21. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  22. 2017Improving Deep Learning using Generic Data Augmentationcited by 36
  23. 2019Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Datacited by 33
  24. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  25. 2021Understanding the Generalization Benefit of Model Invariance from a Data Perspectivecited by 22
  26. 2019Further advantages of data augmentation on convolutional neural networkscited by 19
  27. 2019Does Data Augmentation Lead to Positive Margin?cited by 18
  28. 2020How Data Augmentation affects Optimization for Linear Regressioncited by 15
  29. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  30. 2022A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Momentscited by 14
  31. 2021Training or Architecture? How to Incorporate Invariance in Neural Networkscited by 9
  32. 2021On the Effects of Artificial Data Modificationcited by 2
  33. 2022A Simple Strategy to Provable Invariance via Orbit Mappingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 13; 2024: 9; 2025: 12; 2026: 3.

40 papers.

Sort:

By area:

  1. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  2. 2024Measuring Style Similarity in Diffusion Modelscited by 106
  3. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  4. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  5. 2023Layer-wise Linear Mode Connectivitycited by 22
  6. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  7. 2024You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learningcited by 16
  8. 2022tf.data service: A Case for Disaggregating ML Input Data Processingcited by 15
  9. 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
  10. 2023Understanding the Detrimental Class-level Effects of Data Augmentationcited by 12
  11. 2025Model Merging Scaling Laws in Large Language Modelscited by 10
  12. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
28 more
  1. 2024CoDa: Constrained Generation based Data Augmentation for Low-Resource NLPcited by 9
  2. 2024ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptionscited by 8
  3. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 8
  4. 2025Clustering Properties of Self-Supervised Learningcited by 8
  5. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  6. 2023ADASSM: Adversarial Data Augmentation in Statistical Shape Models From Imagescited by 4
  7. 2023Using and Abusing Equivariancecited by 4
  8. 2023Perspectives on the State and Future of Deep Learning - 2023cited by 4
  9. 2024A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learningcited by 4
  10. 2025Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Datacited by 4
  11. 2025A Principled Framework for Multi-View Contrastive Learningcited by 4
  12. 2022Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learningcited by 3
  13. 2023Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedulecited by 3
  14. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  15. 2023Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Imagescited by 2
  16. 2024Understanding the Role of Invariance in Transfer Learningcited by 1
  17. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  18. 2025Narrowing Class-Wise Robustness Gaps in Adversarial Trainingcited by 1
  19. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1
  20. 2025Dealing with the Evil Twins: Improving Random Augmentation by Addressing Catastrophic Forgetting of Diverse Augmentationscited by 1
  21. 2023Subspace-Configurable Networkscited by 0
  22. 2023RandMSAugment: A Mixed-Sample Augmentation for Limited-Data Scenarioscited by 0
  23. 2025Elliptic Loss Regularizationcited by 0
  24. 2025Equally Critical: Samples, Targets, and Their Mappings in Datasetscited by 0
  25. 2025Expert-guided Clinical Text Augmentation via Query-Based Model Collaborationcited by 0
  26. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  27. 2026Understanding vision transformer robustness through the lens of out-of-distribution detectioncited by 0
  28. 2026The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalizationcited by 0