mlsanityData to 7 Oct 2026

The Implicit Bias of Gradient Descent on Separable Data

arXiv 1710.10345 ↗First posted 27 Oct 2017stat.ML cs.LGcited by 649

Not on the map. Why?

Abstract (arXiv)

We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an infimum at infinity, to multi-class problems, and to training a weight layer in a deep network in a certain restricted setting. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization n more complex models and with other optimization methods.

Key facts. Cited by 649 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 108; 2022: 119; 2023: 102; 2024: 90; 2025: 102; 2026: 127.

649 papers.

Sort:

By area:

  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  3. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  4. 2022Matryoshka Representation Learningcited by 268
  5. 2021Deep learning: a statistical viewpointcited by 239
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  7. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  8. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  9. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  10. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  11. 2022Relative representations enable zero-shot latent space communicationcited by 175
  12. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
38 more
  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  5. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  6. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  8. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  9. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  10. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  11. 2023Explaining grokking through circuit efficiencycited by 105
  12. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  13. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  14. 2022Vision Transformers provably learn spatial structurecited by 92
  15. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  16. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  17. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  18. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  19. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  20. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  21. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  22. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  23. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  24. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  25. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  26. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  27. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  28. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  29. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  30. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  31. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  32. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  33. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  34. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  35. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  36. 2023Transformers as Support Vector Machinescited by 67
  37. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  38. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66

50 most cited of 649 All 649 (JSON)