mlsanityData to 7 Oct 2026

Spectrally-normalized margin bounds for neural networks

arXiv 1706.08498 ↗First posted 26 Jun 2017cs.LG cs.NE stat.MLcited by 681

Not on the map. Why?

Abstract (arXiv)

This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.

Key facts. Cited by 681 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 131; 2022: 112; 2023: 114; 2024: 80; 2025: 124; 2026: 119.

681 papers.

Sort:

By area:

  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  3. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  4. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  5. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  6. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  7. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  8. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  9. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  10. 2021Assessing Generalization of SGD via Disagreementcited by 116
  11. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  12. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
38 more
  1. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  2. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  3. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  4. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  5. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  6. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  9. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  10. 2023A Comprehensive Survey on Deep Graph Representation Learningcited by 72
  11. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  12. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  13. 2022Formal Algorithms for Transformerscited by 70
  14. 2021Skew Orthogonal Convolutionscited by 67
  15. 2021Model Complexity of Deep Learning: A Surveycited by 64
  16. 2021A Theory of PAC Learnability of Partial Concept Classescited by 59
  17. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  18. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 57
  19. 2021Exploring Memorization in Adversarial Trainingcited by 55
  20. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  21. 2023How does over-squashing affect the power of GNNs?cited by 53
  22. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  23. 2021Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformerscited by 49
  24. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  25. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  26. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  27. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  28. 2023How Smooth Is Attention?cited by 44
  29. 2021Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustnesscited by 43
  30. 2022Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodologycited by 43
  31. 2021Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overviewcited by 42
  32. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  33. 2021LEADS: Learning Dynamical Systems that Generalize Across Environmentscited by 39
  34. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  35. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  36. 2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 39
  37. 2021Understanding the role of importance weighting for deep learningcited by 38
  38. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38

50 most cited of 681 All 681 (JSON)