mlsanityData to 7 Oct 2026

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

arXiv 1412.6614 ↗First posted 20 Dec 2014cs.LG cs.AI cs.CVcited by 339

Not on the map. Why?

Abstract (arXiv)

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.

Key facts. Cited by 339 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 67; 2022: 83; 2023: 67; 2024: 56; 2025: 35; 2026: 30.

339 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  4. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  5. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  6. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  7. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  8. 2021Assessing Generalization of SGD via Disagreementcited by 116
  9. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  10. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  11. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  12. 2021On the approximation of functions by tanh neural networkscited by 81
38 more
  1. 2024Robust agents learn causal world modelscited by 81
  2. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  4. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  6. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  7. 2021Model Complexity of Deep Learning: A Surveycited by 64
  8. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  9. 2023On the Foundations of Shortcut Learningcited by 64
  10. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  11. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  12. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  13. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  14. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  15. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  16. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  17. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  18. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  19. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  20. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  21. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  22. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  23. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  24. 2021SGD Generalizes Better Than GD (And Regularization Doesn't Help)cited by 38
  25. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  26. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  27. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  28. 2023Deep networks for system identification: a Surveycited by 35
  29. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  30. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 35
  31. 2021Early-stopped neural networks are consistentcited by 34
  32. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  33. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  34. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  35. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  36. 2023A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learningcited by 32
  37. 2024The Implicit Bias of Adam on Separable Datacited by 32
  38. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31

50 most cited of 339 All 339 (JSON)