# Spectral Norm Regularization for Improving the Generalizability of Deep Learning

arXiv: https://arxiv.org/abs/1705.10941 · HTML: https://mlsanity.com/paper/1705.10941/

## Key facts

- Cited by 182 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2017.

## Built on by (most cited first, 30 of 182)

- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Input Perturbation Reduces Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2301.11706/) (2023)
- [Learning Smooth Neural Functions via Lipschitz Regularization](https://mlsanity.com/paper/2202.08345/) (2022)
- [Data-free Backdoor Removal based on Channel Lipschitzness](https://mlsanity.com/paper/2208.03111/) (2022)
- [PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices](https://mlsanity.com/paper/2211.12562/) (2022)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Contrastive Adapters for Foundation Model Group Robustness](https://mlsanity.com/paper/2207.07180/) (2022)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective](https://mlsanity.com/paper/2210.01787/) (2022)
- [Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons](https://mlsanity.com/paper/2102.05363/) (2021)
- [RoMA: Robust Model Adaptation for Offline Model-based Optimization](https://mlsanity.com/paper/2110.14188/) (2021)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning](https://mlsanity.com/paper/2210.06443/) (2022)
- [AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models](https://mlsanity.com/paper/2410.10912/) (2024)
- [Learning with Hyperspherical Uniformity](https://mlsanity.com/paper/2103.01649/) (2021)
- [Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference](https://mlsanity.com/paper/2402.18512/) (2024)
- [LipsFormer: Introducing Lipschitz Continuity to Vision Transformers](https://mlsanity.com/paper/2304.09856/) (2023)
- [Learning Continually by Spectral Regularization](https://mlsanity.com/paper/2406.06811/) (2024)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [RL Fine-Tuning Heals OOD Forgetting in SFT](https://mlsanity.com/paper/2509.12235/) (2025)
- [Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks](https://mlsanity.com/paper/2104.05097/) (2021)
- [Noise Stability Regularization for Improving BERT Fine-tuning](https://mlsanity.com/paper/2107.04835/) (2021)
- [Weight Clipping for Deep Continual and Reinforcement Learning](https://mlsanity.com/paper/2407.01704/) (2024)
- [Boosting the Certified Robustness of L-infinity Distance Nets](https://mlsanity.com/paper/2110.06850/) (2021)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective](https://mlsanity.com/paper/2404.12008/) (2024)

## Abstract (arXiv)

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.