mlsanityData to 7 Oct 2026

Robust Large Margin Deep Neural Networks

arXiv 1605.08254 ↗First posted 26 May 2016stat.ML cs.LG cs.NEcited by 117

Not on the map. Why?

Abstract (arXiv)

The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization re-parametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED and ImageNet datasets.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 24; 2023: 15; 2024: 16; 2025: 9; 2026: 25.

117 papers.

Sort:

By area:

  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  4. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  5. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  6. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  7. 2023How Smooth Is Attention?cited by 44
  8. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  9. 2021Noisy Recurrent Neural Networkscited by 42
  10. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  11. 2021Discrete-Valued Neural Communicationcited by 39
  12. 2021RoMA: Robust Model Adaptation for Offline Model-based Optimizationcited by 39
38 more
  1. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  2. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  3. 2021Rethinking Graph Auto-Encoder Models for Attributed Graph Clusteringcited by 37
  4. 2023On the Stability of Expressive Positional Encodings for Graphscited by 31
  5. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  6. 2021Noisy Feature Mixupcited by 27
  7. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  8. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  9. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 23
  10. 2023Training robust and generalizable quantum modelscited by 21
  11. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  12. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  13. 2022A PAC-Bayesian Generalization Bound for Equivariant Networkscited by 17
  14. 2021LipBaB: Computing exact Lipschitz constant of ReLU networkscited by 16
  15. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  16. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  17. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  18. 2021Analytical bounds on the local Lipschitz constants of ReLU networkscited by 12
  19. 2022Learning-based Design of Luenberger Observers for Autonomous Nonlinear Systemscited by 11
  20. 2024Investigating the Histogram Loss in Regressioncited by 11
  21. 2024Nuclear Norm Regularization for Deep Learningcited by 11
  22. 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
  23. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  24. 2022On the generalization of learning algorithms that do not convergecited by 9
  25. 2022Curved Representation Space of Vision Transformerscited by 8
  26. 2021Adaptive Prototypical Networks with Label Words and Joint Representation Learning for Few-Shot Relation Classificationcited by 7
  27. 2021Recent Advances in Large Margin Learningcited by 7
  28. 2022Understanding the Evolution of Linear Regions in Deep Reinforcement Learningcited by 7
  29. 2022On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspectivecited by 6
  30. 2023Generalization and Estimation Error Bounds for Model-based Neural Networkscited by 6
  31. 2021Bridging the Gap Between Adversarial Robustness and Optimization Biascited by 5
  32. 2021What training reveals about neural network complexitycited by 5
  33. 2022Exact Spectral Norm Regularization for Neural Networkscited by 5
  34. 2024Maintaining Adversarial Robustness in Continuous Learningcited by 5
  35. 2024Deep neural networks for choice analysis: Enhancing behavioral regularity with gradient regularizationcited by 5
  36. 2021Jacobian Regularization for Mitigating Universal Adversarial Perturbationscited by 4
  37. 2021Neuron with Steady Response Leads to Better Generalizationcited by 4
  38. 2022VC Theoretical Explanation of Double Descentcited by 4

50 most cited of 117 All 117 (JSON)