# Regularization for Deep Learning: A Taxonomy

arXiv: https://arxiv.org/abs/1710.10686 · HTML: https://mlsanity.com/paper/1710.10686/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2017.

## Built on by (most cited first, 30 of 91)

- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Incorporating Domain Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2103.00180/) (2021)
- [Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate](https://mlsanity.com/paper/2402.02769/) (2024)
- [Logic Explained Networks](https://mlsanity.com/paper/2108.05149/) (2021)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization](https://mlsanity.com/paper/2303.05506/) (2023)
- [FedFA: Federated Feature Augmentation](https://mlsanity.com/paper/2301.12995/) (2023)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [NoisyMix: Boosting Model Robustness to Common Corruptions](https://mlsanity.com/paper/2202.01263/) (2022)
- [A Simple Data Augmentation for Feature Distribution Skewed Federated Learning](https://mlsanity.com/paper/2306.09363/) (2023)
- [MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models](https://mlsanity.com/paper/2103.03662/) (2021)
- [Towards Private Learning on Decentralized Graphs with Local Differential Privacy](https://mlsanity.com/paper/2201.09398/) (2022)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)
- [An overview of mixing augmentation methods and augmentation strategies](https://mlsanity.com/paper/2107.09887/) (2021)
- [Computer Vision and Deep Learning for Fish Classification in Underwater Habitats: A Survey](https://mlsanity.com/paper/2203.06951/) (2022)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?](https://mlsanity.com/paper/2504.12883/) (2025)
- [Generalization in Neural Networks: A Broad Survey](https://mlsanity.com/paper/2209.01610/) (2022)
- [FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling](https://mlsanity.com/paper/2303.00586/) (2023)
- [Convergence Rate Analysis of LION](https://mlsanity.com/paper/2411.07724/) (2024)
- [On Neural Network Equivalence Checking using SMT Solvers](https://mlsanity.com/paper/2203.11629/) (2022)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problems](https://mlsanity.com/paper/2202.02710/) (2022)
- [Relating Regularization and Generalization through the Intrinsic Dimension of Activations](https://mlsanity.com/paper/2211.13239/) (2022)

## Abstract (arXiv)

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.