mlsanityData to 14 Oct 2026

Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

arXiv 1905.07325 ↗First posted 17 May 2019stat.ML cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the "constrained path"), relate it to the limit of a "margin path" and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a "lexicographic max-margin solution", and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 10; 2024: 11; 2025: 4; 2026: 6.

53 papers.

Sort:

By area:

  1. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  5. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  6. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  7. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  8. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  9. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  10. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  11. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  12. 2021Understanding the role of importance weighting for deep learningcited by 39
38 more
  1. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  2. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  3. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  4. 2021Properties of the After Kernelcited by 27
  5. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  6. 2023On the Implicit Bias of Adamcited by 27
  7. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  8. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  9. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 19
  10. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  11. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  12. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  13. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  14. 2021Optimal Gradient-based Algorithms for Non-concave Bandit Optimizationcited by 15
  15. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  16. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
  17. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  18. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  19. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
  20. 2021Going Beyond Linear RL: Sample Efficient Neural Function Approximationcited by 9
  21. 2024Implicit Bias of Mirror Flow on Separable Datacited by 9
  22. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 8
  23. 2022Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimizationcited by 7
  24. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  25. 2024The Price of Implicit Bias in Adversarially Robust Generalizationcited by 5
  26. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  27. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  28. 2023Generator Born from Classifiercited by 4
  29. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  30. 2022The Implicit Bias of Gradient Descent on Generalized Gated Linear Networkscited by 3
  31. 2024Stealing Training Graphs from Graph Neural Networkscited by 3
  32. 2021Integral representations of shallow neural network with Rectified Power Unit activation functioncited by 2
  33. 2026Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridgecited by 2
  34. 2026Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Datacited by 2
  35. 2023Applying statistical learning theory to deep learningcited by 1
  36. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  37. 2022From Optimization Dynamics to Generalization Bounds via Lojasiewicz Gradient Inequalitycited by 0
  38. 2022Iterative regularization in classification via hinge loss diagonal descentcited by 0

50 most cited of 53 All 53 (JSON)