mlsanityData to 7 Oct 2026

The Implicit Bias of AdaGrad on Separable Data

arXiv 1906.03559 ↗First posted 9 Jun 2019stat.ML cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

We study the implicit bias of AdaGrad on separable linear classification problems. We show that AdaGrad converges to a direction that can be characterized as the solution of a quadratic optimization problem with the same feasible set as the hard SVM problem. We also give a discussion about how different choices of the hyperparameters of AdaGrad might impact this direction. This provides a deeper understanding of why adaptive methods do not seem to have the generalization ability as good as gradient descent does in practice.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 5; 2024: 2; 2025: 1; 2026: 3.

15 papers.

Sort:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2023Transformers as Support Vector Machinescited by 67
  3. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  4. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  5. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  6. 2024The Implicit Bias of Adam on Separable Datacited by 32
  7. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  8. 2023On the Implicit Bias of Adamcited by 25
  9. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  10. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  11. 2023Understanding the robustness difference between stochastic gradient descent and adaptive gradient methodscited by 8
  12. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
3 more
  1. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  2. 2026Dual Space Preconditioning for Gradient Descent in the Overparameterized Regimecited by 0
  3. 2026Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Datacited by 0