mlsanityData to 14 Oct 2026

Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

arXiv 2002.04486 ↗First posted 11 Feb 2020math.OC cs.LG stat.MLcited by 268

Not on the map. Why?

Abstract (arXiv)

Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activation and confirm the statistical benefits of this implicit bias.

Key facts. Cited by 268 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 55; 2022: 62; 2023: 47; 2024: 43; 2025: 34; 2026: 26.

268 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  5. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  6. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  8. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  9. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  10. 2022Vision Transformers provably learn spatial structurecited by 93
  11. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  12. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
38 more
  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  3. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  4. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  5. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  6. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  7. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  8. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  9. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  10. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  11. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  12. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  13. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  14. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  15. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  16. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  17. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  18. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  19. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  20. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  21. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  22. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  23. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  24. 2021Implicit Regularization in Tensor Factorizationcited by 44
  25. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  26. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  27. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  28. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  29. 2021Understanding the role of importance weighting for deep learningcited by 39
  30. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  31. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
  32. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  33. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  34. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  35. 2024The Implicit Bias of Adam on Separable Datacited by 36
  36. 2021Early-stopped neural networks are consistentcited by 34
  37. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  38. 2021On Margin Maximization in Linear and ReLU Networkscited by 34

50 most cited of 268 All 268 (JSON)