mlsanityData to 7 Oct 2026

Characterizing Implicit Bias in Terms of Optimization Geometry

arXiv 1802.08246 ↗First posted 22 Feb 2018stat.ML cs.LGcited by 259

Not on the map. Why?

Abstract (arXiv)

We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by an algorithm can be characterized in terms of the potential or norm of the optimization geometry, and independently of hyperparameter choices such as step-size and momentum.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 47; 2023: 44; 2024: 50; 2025: 35; 2026: 42.

259 papers.

Sort:

By area:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  3. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  4. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  5. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  6. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  7. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  9. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  10. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  11. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  12. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
38 more
  1. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  2. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  3. 2023Transformers as Support Vector Machinescited by 67
  4. 2021Dissecting Supervised Contrastive Learningcited by 64
  5. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  6. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  7. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  8. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  9. 2021What can linearized neural networks actually say about generalization?cited by 49
  10. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  11. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  12. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  13. 2021Implicit Regularization in Tensor Factorizationcited by 43
  14. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  15. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  16. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  17. 2021SGD Generalizes Better Than GD (And Regularization Doesn't Help)cited by 38
  18. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  19. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  20. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  21. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  22. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  23. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  24. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  25. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  26. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  27. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  28. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  29. 2024The Implicit Bias of Adam on Separable Datacited by 32
  30. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  31. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  32. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  33. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  34. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  35. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  36. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  37. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  38. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 25

50 most cited of 259 All 259 (JSON)