mlsanityData to 7 Oct 2026

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks

arXiv 2410.22069 ↗First posted 29 Oct 2024cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy, and (b) any limit point of the training trajectory corresponds to a KKT point of the corresponding margin-maximization problem. We experimentally zoom into the trajectories of neural networks optimized with various steepest descent algorithms, highlighting connections to the implicit bias of popular adaptive methods (Adam and Shampoo).

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

30 of 57 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Convex optimizationcited by 2,479
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  9. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2024Old Optimizer, New Norm: An Anthologycited by 173
  12. 2020Directional convergence and alignment in deep learningcited by 172
18 more
  1. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  2. 2018Stochastic subgradient method converges on tame functionscited by 120
  3. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  4. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2024Scalable Optimization in the Modular Normcited by 64
  8. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  9. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  10. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  11. 2022The Implicit Bias of Benign Overfittingcited by 38
  12. 2013Margins, Shrinkage, and Boostingcited by 35
  13. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  14. 2024The Implicit Bias of Adam on Separable Datacited by 32
  15. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  16. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  17. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  18. 2024The Price of Implicit Bias in Adversarially Robust Generalizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

  1. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  2. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 25
  3. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  4. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  5. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  6. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  7. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  8. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  9. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 0
  10. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  11. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  12. 2026Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamicscited by 0
1 more
  1. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0