mlsanityData to 7 Oct 2026

Regularization for Deep Learning: A Taxonomy

arXiv 1710.10686 ↗First posted 29 Oct 2017cs.LG cs.AI cs.CVcited by 91

Not on the map. Why?

Abstract (arXiv)

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 20; 2023: 20; 2024: 14; 2025: 11; 2026: 3.

91 papers.

Sort:

By area:

  1. 2022Long-Tailed Recognition via Weight Balancingcited by 107
  2. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  3. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  4. 2021Incorporating Domain Knowledge into Deep Neural Networkscited by 56
  5. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  6. 2021Logic Explained Networkscited by 45
  7. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  8. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  9. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  10. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  11. 2023FedFA: Federated Feature Augmentationcited by 22
  12. 2024Video Anomaly Detection in 10 Years: A Survey and Outlookcited by 18
38 more
  1. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  2. 2022NoisyMix: Boosting Model Robustness to Common Corruptionscited by 17
  3. 2021MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world modelscited by 16
  4. 2022Towards Private Learning on Decentralized Graphs with Local Differential Privacycited by 16
  5. 2023A Simple Data Augmentation for Feature Distribution Skewed Federated Learningcited by 16
  6. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  7. 2021An overview of mixing augmentation methods and augmentation strategiescited by 14
  8. 2022Computer Vision and Deep Learning for Fish Classification in Underwater Habitats: A Surveycited by 14
  9. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  10. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  11. 2022Generalization in Neural Networks: A Broad Surveycited by 11
  12. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  13. 2024Convergence Rate Analysis of LIONcited by 11
  14. 2022On Neural Network Equivalence Checking using SMT Solverscited by 10
  15. 2024Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Surveycited by 10
  16. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  17. 2022Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problemscited by 9
  18. 2022Relating Regularization and Generalization through the Intrinsic Dimension of Activationscited by 9
  19. 2021Parameterized Hypercomplex Graph Neural Networks for Graph Classificationcited by 7
  20. 2021Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutionscited by 7
  21. 2022Common Phone: A Multilingual Dataset for Robust Acoustic Modellingcited by 7
  22. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  23. 2023Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognitioncited by 7
  24. 2023fairret: a Framework for Differentiable Fairness Regularization Termscited by 7
  25. 2021KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learningcited by 6
  26. 2024Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfittingcited by 6
  27. 2021Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Datacited by 5
  28. 2023Robust Losses for Decision-Focused Learningcited by 5
  29. 2021Learning distinct features helps, provablycited by 4
  30. 2021Encoder-Decoder Architectures for Clinically Relevant Coronary Artery Segmentationcited by 4
  31. 2022Optimal Activation Functions for the Random Features Regression Modelcited by 4
  32. 2022Applications of Deep Learning in Fish Habitat Monitoring: A Tutorial and Surveycited by 4
  33. 2023WLD-Reg: A Data-dependent Within-layer Diversity Regularizercited by 4
  34. 2023Hard-aware Instance Adaptive Self-training for Unsupervised Cross-domain Semantic Segmentationcited by 4
  35. 2023Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networkscited by 4
  36. 2023Sparse then Prune: Toward Efficient Vision Transformerscited by 4
  37. 2025TIF: Learning Temporal Invariance in Android Malware Detectorscited by 4
  38. 2025Position: Certified Robustness Does Not (Yet) Imply Model Securitycited by 4

50 most cited of 91 All 91 (JSON)