# Data augmentation instead of explicit regularization

arXiv: https://arxiv.org/abs/1806.03852 · HTML: https://mlsanity.com/paper/1806.03852/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2018.

## Built on by (most cited first, 30 of 63)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Fair Mixup: Fairness via Interpolation](https://mlsanity.com/paper/2103.06503/) (2021)
- [Dataset Condensation with Contrastive Signals](https://mlsanity.com/paper/2202.02916/) (2022)
- [Size-Invariant Graph Representations for Graph Classification Extrapolations](https://mlsanity.com/paper/2103.05045/) (2021)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2505.20075/) (2025)
- [SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries](https://mlsanity.com/paper/2302.12828/) (2023)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting](https://mlsanity.com/paper/2206.09450/) (2022)
- [Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification](https://mlsanity.com/paper/2109.00523/) (2021)
- [A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments](https://mlsanity.com/paper/2202.08325/) (2022)
- [When to Learn What: Model-Adaptive Data Augmentation Curriculum](https://mlsanity.com/paper/2309.04747/) (2023)
- [Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation](https://mlsanity.com/paper/2206.01909/) (2022)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization](https://mlsanity.com/paper/2202.01832/) (2022)
- [Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning](https://mlsanity.com/paper/2602.11149/) (2026)
- [Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers](https://mlsanity.com/paper/2405.03730/) (2024)
- [The Effects of Hallucinations in Synthetic Training Data for Relation Extraction](https://mlsanity.com/paper/2410.08393/) (2024)
- [A trainable monogenic ConvNet layer robust in front of large contrast changes in image classification](https://mlsanity.com/paper/2109.06926/) (2021)
- [Unleashing the Potential of Regularization Strategies in Learning with Noisy Labels](https://mlsanity.com/paper/2307.05025/) (2023)
- [Fast Jacobian-Vector Product for Deep Networks](https://mlsanity.com/paper/2104.00219/) (2021)
- [GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory](https://mlsanity.com/paper/2402.05916/) (2024)
- [Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup](https://mlsanity.com/paper/2109.07177/) (2021)
- [Augmenting the User-Item Graph with Textual Similarity Models](https://mlsanity.com/paper/2109.09358/) (2021)
- [Training Dynamics of Deep Network Linear Regions](https://mlsanity.com/paper/2310.12977/) (2023)
- [Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks](https://mlsanity.com/paper/2509.24886/) (2025)
- [Automated Cleanup of the ImageNet Dataset by Model Consensus, Explainability and Confident Learning](https://mlsanity.com/paper/2103.16324/) (2021)

## Abstract (arXiv)

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.