Data to 14 Oct 2026

Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

arXiv 1906.05890 ↗First posted 13 Jun 2019cs.LG cs.NE stat.MLcited by 320

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time. We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem. Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks. We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets. Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.

Key facts. Cited by 320 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 55; 2023: 56; 2024: 56; 2025: 46; 2026: 66.

320 papers.

Sort:

By area:

2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
2022Reconstructing Training Data from Trained Neural Networkscited by 133
2021Can contrastive learning avoid shortcut solutions?cited by 114
2023Explaining grokking through circuit efficiencycited by 111
2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
2022Vision Transformers provably learn spatial structurecited by 93
2022Learning Single-Index Models with Shallow Neural Networkscited by 92
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91

38 more

2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
2023Transformers as Support Vector Machinescited by 74
2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
2023Max-Margin Token Selection in Attention Mechanismcited by 58
2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
2022C-Mixup: Improving Generalization in Regressioncited by 57
2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
2022Mechanistic Mode Connectivitycited by 50
2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
2025Grokking at the Edge of Numerical Stabilitycited by 46
2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
2021Noisy Recurrent Neural Networkscited by 42
2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
2022Towards understanding how momentum improves generalization in deep learningcited by 40
2021Understanding the role of importance weighting for deep learningcited by 39
2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
2024The Implicit Bias of Adam on Separable Datacited by 36
2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
2021Early-stopped neural networks are consistentcited by 34

50 most cited of 320 All 320 (JSON)