mlsanityData to 7 Oct 2026

On Margin Maximization in Linear and ReLU Networks

arXiv 2110.02732 ↗First posted 6 Oct 2021cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in the parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

31 of 57 references.

References · 31

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2017Exploring Generalization in Deep Learningcited by 595
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  10. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  11. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  12. 2018Gradient descent aligns the layers of deep linear networkscited by 182
19 more
  1. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  2. 2020Directional convergence and alignment in deep learningcited by 172
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  4. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  5. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  6. 2018Convergence of Gradient Descent on Separable Datacited by 108
  7. 2018Risk and parameter convergence of logistic regressioncited by 81
  8. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  9. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  10. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  11. 2020Gradient descent follows the regularization path for general lossescited by 57
  12. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  13. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  14. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  15. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  16. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  17. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  18. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  19. 2020On implicit regularization: Morse functions and applications to matrix factorizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 8; 2024: 5; 2025: 4; 2026: 5.

34 papers.

Sort:

By area:

  1. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  4. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  5. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  6. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  7. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  8. 2024The Implicit Bias of Adam on Separable Datacited by 32
  9. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  10. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  11. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  12. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
22 more
  1. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  2. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  3. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  4. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  5. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  6. 2022Feature selection with gradient descent on two-layer networks in low-rotation regimescited by 13
  7. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 13
  8. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  9. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  10. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
  11. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  12. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  13. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  14. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  15. 2026Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridgecited by 2
  16. 2022Improved Convergence Guarantees for Shallow Neural Networkscited by 1
  17. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  18. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1
  19. 2026Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networkscited by 0
  20. 2026Estimating Implicit Regularization in Deep Learningcited by 0
  21. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  22. 2026Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Datacited by 0