mlsanityData to 7 Oct 2026

Sample Efficiency of Data Augmentation Consistency Regularization

arXiv 2202.12230 ↗First posted 24 Feb 2022cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data. In this paper, we take a step in this direction - we first present a simple and novel analysis for linear regression with label invariant augmentations, demonstrating that data augmentation consistency (DAC) is intrinsically more efficient than empirical risk minimization on augmented data (DA-ERM). The analysis is then extended to misspecified augmentations (i.e., augmentations that change the labels), which again demonstrates the merit of DAC over DA-ERM. Further, we extend our analysis to non-linear models (e.g., neural networks) and present generalization bounds. Finally, we perform experiments that make a clean and apples-to-apples comparison (i.e., with no extra modeling or data tweaks) between DAC and DA-ERM using CIFAR-100 and WideResNet; these together demonstrate the superior efficacy of DAC.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

39 of 59 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016Wide Residual Networkscited by 3,057
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  12. 2016Group Equivariant Convolutional Networkscited by 1,154
27 more
  1. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  2. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  3. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  4. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  5. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  6. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  7. 2019Fast AutoAugmentcited by 243
  8. 2014Learning with Pseudo-Ensemblescited by 241
  9. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  10. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  11. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  12. 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
  13. 2019Scale-Equivariant Steerable Networkscited by 110
  14. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  15. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  16. 2016Rotation equivariant vector field networkscited by 98
  17. 2021Learning with invariances in random features and kernel modelscited by 89
  18. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  19. 2017Oriented Response Networkscited by 67
  20. 2020Learning Invariances in Neural Networkscited by 52
  21. 2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
  22. 2020On the Generalization Effects of Linear Transformations in Data Augmentationcited by 47
  23. 2019Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Datacited by 33
  24. 2022A Theory of PAC Learnability under Transformation Invariancescited by 21
  25. 2019Does Data Augmentation Lead to Positive Margin?cited by 18
  26. 2020Functional Regularization for Representation Learning: A Unified Theoretical Perspectivecited by 18
  27. 2018Efficient Augmentation via Data Subsamplingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 6; 2024: 6; 2025: 4; 2026: 6.

28 papers.

Sort:

By area:

  1. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  2. 2023Instruction Tuned Models are Quick Learnerscited by 21
  3. 2022Data Augmentation as Feature Manipulationcited by 18
  4. 2022Provably Learning Diverse Features in Multi-View Data with Midpoint Mixupcited by 14
  5. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  6. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  7. 2022Regularising for invariance to data augmentation improves supervised learningcited by 9
  8. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clusteringcited by 7
  9. 2023DAC-MR: Data Augmentation Consistency Based Meta-Regularization for Meta-Learningcited by 4
  10. 2024Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?cited by 4
  11. 2022Adaptively Weighted Data Augmentation Consistency Regularization for Robust Optimization under Concept Shiftcited by 3
  12. 2023Understanding Data Augmentation from a Robustness Perspectivecited by 3
16 more
  1. 2024For Better or For Worse? Learning Minimum Variance Features With Label Augmentationcited by 3
  2. 2025Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networkscited by 3
  3. 2026Data Augmentation: A Fourier Analysis Perspectivecited by 3
  4. 2023LEDetection: A Simple Framework for Semi-Supervised Few-Shot Object Detectioncited by 2
  5. 2024Bridging Domains with Approximately Shared Featurescited by 2
  6. 2026Selective Cross-View Consistency for World Action Models: Held-Out Viewpoint Robustness Without Test-Time Camera Informationcited by 1
  7. 2022MixBoost: Improving the Robustness of Deep Neural Networks by Boosting Data Augmentationcited by 0
  8. 2023Randomized Dimension Reduction with Statistical Guaranteescited by 0
  9. 2024A semi-supervised learning using over-parameterized regressioncited by 0
  10. 2024Unsupervised Domain Adaptation for Action Recognition via Self-Ensembling and Conditional Embedding Alignmentcited by 0
  11. 2025Elliptic Loss Regularizationcited by 0
  12. 2025Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performancecited by 0
  13. 2026The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalizationcited by 0
  14. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0
  15. 2026Fast Rates for Semi-Supervised Learning via Data-Augmentation Graph Regularizationcited by 0
  16. 2026Sparse Data Augmentation for Optimization with Provable Guaranteescited by 0