mlsanityData to 7 Oct 2026

On the Implicit Bias in Deep-Learning Algorithms

arXiv 2208.12591 ↗First posted 26 Aug 2022cs.LG stat.MLcited by 84

Not on the map. Why?

Abstract (arXiv)

Gradient-based deep-learning algorithms exhibit remarkable performance in practice, but it is not well-understood why they are able to generalize despite having more parameters than training examples. It is believed that implicit bias is a key factor in their ability to generalize, and hence it was widely studied in recent years. In this short survey, we explain the notion of implicit bias, review main results and discuss their implications.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 72 of its 87 references that are in the map. First posted on arXiv 26 Aug 2022.

Builds on

72 of 87 references.

References · 72

  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
38 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  2. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  3. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  13. 2020Directional convergence and alignment in deep learningcited by 172
  14. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  15. 2020Implicit Gradient Regularizationcited by 145
  16. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  17. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  18. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  19. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  20. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  21. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  22. 2018Convergence of Gradient Descent on Separable Datacited by 108
  23. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  24. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  25. 2019How do infinite width bounded norm networks look in function space?cited by 105
  26. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  27. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  28. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  29. 2022Understanding the unstable convergence of gradient descentcited by 84
  30. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  31. 2018Risk and parameter convergence of logistic regressioncited by 81
  32. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  33. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  34. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  35. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  36. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  37. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  38. 2020Gradient descent follows the regularization path for general lossescited by 57

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 23; 2025: 29; 2026: 10.

84 papers.

Sort:

By area:

  1. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  2. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  3. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  4. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  5. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  6. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  7. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  8. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  9. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  10. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  11. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  12. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
38 more
  1. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  2. 2025Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptivecited by 20
  3. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  4. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  5. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  6. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  7. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  8. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  9. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  10. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  11. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  12. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  13. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  14. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  15. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  16. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  17. 2024The Implicit Bias of Gradient Descent on Separable Multiclass Datacited by 11
  18. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  19. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 10
  20. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  21. 2025An entropy formula for the Deep Linear Networkcited by 8
  22. 2023On the Implicit Bias of Linear Equivariant Steerable Networkscited by 7
  23. 2025Deep Unfolding: Recent Developments, Theory, and Design Guidelinescited by 7
  24. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  25. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  26. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  27. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  28. 2024The Price of Implicit Bias in Adversarially Robust Generalizationcited by 4
  29. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  30. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  31. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  32. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  33. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 2
  34. 2024Causal hybrid modeling with double machine learningcited by 2
  35. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  36. 2025Bigger Isn't Always Better: Towards a General Prior for Medical Image Reconstructioncited by 2
  37. 2025Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fieldscited by 2
  38. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2

50 most cited of 84 All 84 (JSON)