mlsanityData to 7 Oct 2026

Implicit Regularization in Deep Learning

arXiv 1709.01953 ↗First posted 6 Sep 2017cs.LGcited by 90

Not on the map. Why?

Abstract (arXiv)

In an attempt to better understand generalization in deep learning, we study several possible explanations. We show that implicit regularization induced by the optimization method is playing a key role in generalization and success of deep learning models. Motivated by this view, we study how different complexity measures can ensure generalization and explain how optimization algorithms can implicitly regularize complexity measures. We empirically investigate the ability of these measures to explain different observed phenomena in deep learning. We further study the invariances in neural networks, suggest complexity measures and optimization algorithms that have similar invariances to those in neural networks and evaluate them on a number of learning tasks.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 22; 2023: 21; 2024: 18; 2025: 11; 2026: 8.

90 papers.

Sort:

By area:

  1. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  2. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  3. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  4. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  6. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  7. 2021Model Complexity of Deep Learning: A Surveycited by 64
  8. 2022C-Mixup: Improving Generalization in Regressioncited by 57
  9. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  10. 2021Consensus Control for Decentralized Deep Learningcited by 49
  11. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  12. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
38 more
  1. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  2. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 35
  3. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  4. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  5. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  6. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  7. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  8. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  9. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  10. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  11. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  12. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  13. 2022A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Momentscited by 14
  14. 2022Robust Constrained Reinforcement Learningcited by 14
  15. 2022PAC-Bayesian Domain Adaptation Bounds for Multiclass Learnerscited by 13
  16. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  17. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  18. 2021The Dynamics of Gradient Descent for Overparametrized Neural Networkscited by 11
  19. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  20. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  21. 2022DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learningcited by 10
  22. 2023A path-norm toolkit for modern networks: consequences, promises and challengescited by 10
  23. 2024Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Surveycited by 10
  24. 2025DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretationcited by 9
  25. 2021Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximationcited by 8
  26. 2023Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networkscited by 8
  27. 2023PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learningcited by 8
  28. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  29. 2025Shape of Thought: When Distribution Matters More than Correctness in Reasoning Taskscited by 8
  30. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  31. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  32. 2023Consistent Optimal Transport with Empirical Conditional Measurescited by 6
  33. 2022Controlling the Complexity and Lipschitz Constant improves polynomial netscited by 5
  34. 2025Reason to Rote: Rethinking Memorization in Reasoningcited by 5
  35. 2021From inexact optimization to learning via gradient concentrationcited by 4
  36. 2022Learning Temporal Resolution in Spectrogram for Audio Classificationcited by 4
  37. 2023Exploring the Complexity of Deep Neural Networks through Functional Equivalencecited by 4
  38. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4

50 most cited of 90 All 90 (JSON)