mlsanityData to 14 Oct 2026

Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

arXiv 2608.04382 ↗First posted 5 Aug 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within O(exp(exp(-δ))) iterations, where δ>0 is the permissible alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

33 of 38 references.

References · 33

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  6. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  10. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  11. 2018Convergence of Gradient Descent on Separable Datacited by 110
  12. 2019How do infinite width bounded norm networks look in function space?cited by 106
21 more
  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  2. 2022Understanding the unstable convergence of gradient descentcited by 86
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  4. 2020Gradient descent follows the regularization path for general lossescited by 59
  5. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  6. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  7. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  8. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  9. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
  10. 2024The Implicit Bias of Adam on Separable Datacited by 36
  11. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  12. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  13. 2022Stability vs Implicit Bias of Gradient Methods on Separable Data and Beyondcited by 29
  14. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  15. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  16. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
  17. 2024The Implicit Bias of Gradient Descent on Separable Multiclass Datacited by 11
  18. 2024Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regressioncited by 6
  19. 2024From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizescited by 6
  20. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  21. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1

Built on by

Not cited yet.