mlsanityData to 14 Oct 2026

Explicit regularization and implicit bias in deep network classifiers trained with the square loss

arXiv 2101.00072 ↗First posted 31 Dec 2020cs.LG stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

Deep ReLU networks trained with the square loss have been observed to perform well in classification tasks. We provide here a theoretical justification based on analysis of the associated gradient flow. We show that convergence to a solution with the absolute minimum norm is expected when normalization techniques such as Batch Normalization (BN) or Weight Normalization (WN) are used together with Weight Decay (WD). The main property of the minimizers that bounds their expected error is the norm: we prove that among all the close-to-interpolating solutions, the ones associated with smaller Frobenius norms of the unnormalized weight matrices have better margin and better bounds on the expected classification error. With BN but in the absence of WD, the dynamical system is singular. Implicit dynamical regularization -- that is zero-initial conditions biasing the dynamics towards high margin solutions -- is also possible in the no-BN and no-WD case. The theory yields several predictions, including the role of BN and weight decay, aspects of Papyan, Han and Donoho's Neural Collapse and the constraints induced by BN on the network weights.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 8; 2023: 12; 2024: 9; 2025: 3.

37 papers.

Sort:

  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  3. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  4. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  5. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  6. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  7. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  8. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  9. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  10. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  11. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  12. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
25 more
  1. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  2. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  3. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  4. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  5. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  6. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  7. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  8. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  9. 2023Neural (Tangent Kernel) Collapsecited by 18
  10. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  11. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  12. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  13. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  14. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  15. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
  16. 2023A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalizationcited by 7
  17. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 7
  18. 2024Pushing Boundaries: Mixup's Influence on Neural Collapsecited by 7
  19. 2025Consistent Supervised-Unsupervised Alignment for Generalized Category Discoverycited by 6
  20. 2023On the Dynamics Under the Unhinged Loss and Beyondcited by 3
  21. 2022Memorization-Dilation: Modeling Neural Collapse Under Label Noisecited by 2
  22. 2023It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Modelscited by 1
  23. 2025SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapsecited by 1
  24. 2024Graph Classification with GNNs: Optimisation, Representation and Inductive Biascited by 0
  25. 2025Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapsecited by 0