# Stable Rank Normalization for Improved Generalization in Neural Networks and GANs

arXiv: https://arxiv.org/abs/1906.04659 · HTML: https://mlsanity.com/paper/1906.04659/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 30 of 41)

- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [What Makes and Breaks Safety Fine-tuning? A Mechanistic Study](https://mlsanity.com/paper/2407.10264/) (2024)
- [Implicit Regularization Towards Rank Minimization in ReLU Networks](https://mlsanity.com/paper/2201.12760/) (2022)
- [Rethinking Positional Encoding](https://mlsanity.com/paper/2107.02561/) (2021)
- [An Impartial Take to the CNN vs Transformer Robustness Contest](https://mlsanity.com/paper/2207.11347/) (2022)
- [Noise Stability Regularization for Improving BERT Fine-tuning](https://mlsanity.com/paper/2107.04835/) (2021)
- [RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness](https://mlsanity.com/paper/2206.14502/) (2022)
- [Deep Learning Meets Sparse Regularization: A Signal Processing Perspective](https://mlsanity.com/paper/2301.09554/) (2023)
- [Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training](https://mlsanity.com/paper/2312.00359/) (2023)
- [Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations](https://mlsanity.com/paper/2504.05586/) (2025)
- [PoLAR: Polar-Decomposed Low-Rank Adapter Representation](https://mlsanity.com/paper/2506.03133/) (2025)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning](https://mlsanity.com/paper/2506.00772/) (2025)
- [Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias](https://mlsanity.com/paper/2506.06280/) (2025)
- [Self-Supervised GANs with Label Augmentation](https://mlsanity.com/paper/2106.08601/) (2021)
- [Towards Practical Control of Singular Values of Convolutional Layers](https://mlsanity.com/paper/2211.13771/) (2022)
- [Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation](https://mlsanity.com/paper/2406.19049/) (2024)
- [Subspace-Boosted Model Merging](https://mlsanity.com/paper/2506.16506/) (2025)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Optimizing Rank for High-Fidelity Implicit Neural Representations](https://mlsanity.com/paper/2512.14366/) (2025)
- [Spectral Tensor Train Parameterization of Deep Learning Layers](https://mlsanity.com/paper/2103.04217/) (2021)
- [Stable Long-Term Recurrent Video Super-Resolution](https://mlsanity.com/paper/2112.08950/) (2021)
- [Probabilistic Lipschitzness and the Stable Rank for Comparing Explanation Models](https://mlsanity.com/paper/2402.18863/) (2024)
- [On the Impact of Stable Ranks in Deep Nets](https://mlsanity.com/paper/2110.02333/) (2021)
- [Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds](https://mlsanity.com/paper/2211.10558/) (2022)
- [Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes](https://mlsanity.com/paper/2507.00327/) (2025)
- [Frayed RoPE and Long Inputs: A Geometric Perspective](https://mlsanity.com/paper/2603.18017/) (2026)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)
- [Meta-free few-shot learning via representation learning with weight averaging](https://mlsanity.com/paper/2204.12466/) (2022)

## Abstract (arXiv)

Exciting new work on the generalization bounds for neural networks (NN) given by Neyshabur et al. , Bartlett et al. closely depend on two parameter-depenedent quantities: the Lipschitz constant upper-bound and the stable rank (a softer version of the rank operator). This leads to an interesting question of whether controlling these quantities might improve the generalization behaviour of NNs. To this end, we propose stable rank normalization (SRN), a novel, optimal, and computationally efficient weight-normalization scheme which minimizes the stable rank of a linear operator. Surprisingly we find that SRN, inspite of being non-convex problem, can be shown to have a unique optimal solution. Moreover, we show that SRN allows control of the data-dependent empirical Lipschitz constant, which in contrast to the Lipschitz upper-bound, reflects the true behaviour of a model on a given dataset. We provide thorough analyses to show that SRN, when applied to the linear layers of a NN for classification, provides striking improvements-11.3% on the generalization gap compared to the standard NN along with significant reduction in memorization. When applied to the discriminator of GANs (called SRN-GAN) it improves Inception, FID, and Neural divergence scores on the CIFAR 10/100 and CelebA datasets, while learning mappings with low empirical Lipschitz constants.