# Robust Large Margin Deep Neural Networks

arXiv: https://arxiv.org/abs/1605.08254 · HTML: https://mlsanity.com/paper/1605.08254/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2016.

## Built on by (most cited first, 30 of 118)

- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [How Smooth Is Attention?](https://mlsanity.com/paper/2312.14820/) (2023)
- [UnitedQA: A Hybrid Approach for Open Domain Question Answering](https://mlsanity.com/paper/2101.00178/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons](https://mlsanity.com/paper/2102.05363/) (2021)
- [Discrete-Valued Neural Communication](https://mlsanity.com/paper/2107.02367/) (2021)
- [RoMA: Robust Model Adaptation for Offline Model-based Optimization](https://mlsanity.com/paper/2110.14188/) (2021)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System](https://mlsanity.com/paper/2303.10837/) (2023)
- [Rethinking Graph Auto-Encoder Models for Attributed Graph Clustering](https://mlsanity.com/paper/2107.08562/) (2021)
- [On the Stability of Expressive Positional Encodings for Graphs](https://mlsanity.com/paper/2310.02579/) (2023)
- [Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks](https://mlsanity.com/paper/2104.05097/) (2021)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [Robustness Implies Generalization via Data-Dependent Generalization Bounds](https://mlsanity.com/paper/2206.13497/) (2022)
- [Training robust and generalizable quantum models](https://mlsanity.com/paper/2311.11871/) (2023)
- [Nearest Class-Center Simplification through Intermediate Layers](https://mlsanity.com/paper/2201.08924/) (2022)
- [A PAC-Bayesian Generalization Bound for Equivariant Networks](https://mlsanity.com/paper/2210.13150/) (2022)
- [On the Equivalence between Neural Network and Support Vector Machine](https://mlsanity.com/paper/2111.06063/) (2021)
- [LipBaB: Computing exact Lipschitz constant of ReLU networks](https://mlsanity.com/paper/2105.05495/) (2021)
- [Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/2108.05623/) (2021)
- [Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation](https://mlsanity.com/paper/2405.17484/) (2024)
- [Instance-Dependent Generalization Bounds via Optimal Transport](https://mlsanity.com/paper/2211.01258/) (2022)
- [Analytical bounds on the local Lipschitz constants of ReLU networks](https://mlsanity.com/paper/2104.14672/) (2021)

## Abstract (arXiv)

The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization re-parametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED and ImageNet datasets.