mlsanityData to 14 Oct 2026

The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

arXiv 2210.03820 ↗First posted 7 Oct 2022cs.LG stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

In this work, we explore the maximum-margin bias of quasi-homogeneous neural networks trained with gradient flow on an exponential loss and past a point of separability. We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogeneous activations, even those with biases, residual connections, and normalization layers, while structured enough to enable geometric analysis of its gradient dynamics. Using this analysis, we generalize the existing results of maximum-margin bias for homogeneous networks to this richer class of models. We find that gradient flow implicitly favors a subset of the parameters, unlike in the case of a homogeneous model where all parameters are treated equally. We demonstrate through simple examples how this strong favoritism toward minimizing an asymmetric norm can degrade the robustness of quasi-homogeneous models. On the other hand, we conjecture that this norm-minimization discards, when possible, unnecessary higher-order parameters, reducing the model to a sparser parameterization. Lastly, by applying our theorem to sufficiently expressive neural networks with normalization layers, we reveal a universal mechanism behind the empirical phenomenon of Neural Collapse.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

28 of 45 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  7. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  8. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  9. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  10. 2020Directional convergence and alignment in deep learningcited by 178
  11. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  12. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
16 more
  1. 2020Neural collapse with unconstrained featurescited by 142
  2. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  3. 2018Stochastic subgradient method converges on tame functionscited by 124
  4. 2018Convergence of Gradient Descent on Separable Datacited by 110
  5. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  7. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  9. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  10. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  11. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  12. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  13. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  14. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  15. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  16. 2018When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 9; 2025: 4; 2026: 3.

25 papers.

Sort:

  1. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  2. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  3. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  4. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  5. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  6. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  7. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  8. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  9. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  10. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 20
  11. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  12. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
13 more
  1. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  2. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  3. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  4. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 11
  5. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  6. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  7. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  8. 2023Generator Born from Classifiercited by 4
  9. 2026The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networkscited by 2
  10. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  11. 2023Hierarchical Simplicity Bias of Neural Networkscited by 0
  12. 2025Embedding principle of homogeneous neural network for classification problemcited by 0
  13. 2026Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networkscited by 0