# Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data

arXiv: https://arxiv.org/abs/1909.09148 · HTML: https://mlsanity.com/paper/1909.09148/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [SSMix: Saliency-Based Span Mixup for Text Classification](https://mlsanity.com/paper/2106.08062/) (2021)
- [Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective](https://mlsanity.com/paper/2111.03874/) (2021)
- [How Re-sampling Helps for Long-Tail Learning?](https://mlsanity.com/paper/2310.18236/) (2023)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Direct Differentiable Augmentation Search](https://mlsanity.com/paper/2104.04282/) (2021)
- [Sample Efficiency of Data Augmentation Consistency Regularization](https://mlsanity.com/paper/2202.12230/) (2022)
- [Out-of-Domain Robustness via Targeted Augmentations](https://mlsanity.com/paper/2302.11861/) (2023)
- [Pure Noise to the Rescue of Insufficient Data: Improving Imbalanced Classification by Training on Random Noise Images](https://mlsanity.com/paper/2112.08810/) (2021)
- [Mixup Without Hesitation](https://mlsanity.com/paper/2101.04342/) (2021)
- [DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain](https://mlsanity.com/paper/2410.12307/) (2024)
- [OOD Augmentation May Be at Odds with Open-Set Recognition](https://mlsanity.com/paper/2206.04242/) (2022)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation](https://mlsanity.com/paper/2410.16020/) (2024)
- [Jitter: Random Jittering Loss Function](https://mlsanity.com/paper/2106.13749/) (2021)
- [SETA: Semantic-Aware Token Augmentation for Domain Generalization](https://mlsanity.com/paper/2403.11792/) (2024)
- [Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer](https://mlsanity.com/paper/2407.18554/) (2024)
- [Octave Mix: Data augmentation using frequency decomposition for activity recognition](https://mlsanity.com/paper/2101.02882/) (2021)
- [Theoretically Motivated Data Augmentation and Regularization for Portfolio Construction](https://mlsanity.com/paper/2106.04114/) (2021)
- [SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation](https://mlsanity.com/paper/2212.07072/) (2022)
- [Learning from Reduced Labels for Long-Tailed Data](https://mlsanity.com/paper/2403.16469/) (2024)
- [Data augmentation and feature selection for automatic model recommendation in computational physics](https://mlsanity.com/paper/2101.04530/) (2021)
- [Swapping Semantic Contents for Mixing Images](https://mlsanity.com/paper/2205.10158/) (2022)
- [Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture](https://mlsanity.com/paper/2207.10351/) (2022)
- [Dealing with the Evil Twins: Improving Random Augmentation by Addressing Catastrophic Forgetting of Diverse Augmentations](https://mlsanity.com/paper/2506.08240/) (2025)
- [Sample-efficient Adversarial Imitation Learning](https://mlsanity.com/paper/2303.07846/) (2023)
- [Deep Active Learning with Structured Neural Depth Search](https://mlsanity.com/paper/2306.02808/) (2023)
- [Randomized Dimension Reduction with Statistical Guarantees](https://mlsanity.com/paper/2310.01739/) (2023)

## Abstract (arXiv)

Data augmentation has been widely applied as an effective methodology to improve generalization in particular when training deep neural networks. Recently, researchers proposed a few intensive data augmentation techniques, which indeed improved accuracy, yet we notice that these methods augment data have also caused a considerable gap between clean and augmented data. In this paper, we revisit this problem from an analytical perspective, for which we estimate the upper-bound of expected risk using two terms, namely, empirical risk and generalization error, respectively. We develop an understanding of data augmentation as regularization, which highlights the major features. As a result, data augmentation significantly reduces the generalization error, but meanwhile leads to a slightly higher empirical risk. On the assumption that data augmentation helps models converge to a better region, the model can benefit from a lower empirical risk achieved by a simple method, i.e., using less-augmented data to refine the model trained on fully-augmented data. Our approach achieves consistent accuracy gain on a few standard image classification benchmarks, and the gain transfers to object detection.