# Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

arXiv: https://arxiv.org/abs/2009.02773 · HTML: https://mlsanity.com/paper/2009.02773/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2020.

## Built on by (most cited first, 21 of 21)

- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning](https://mlsanity.com/paper/2405.15624/) (2024)
- [Hyperbolic Deep Reinforcement Learning](https://mlsanity.com/paper/2210.01542/) (2022)
- [On the Privacy Properties of GAN-generated Samples](https://mlsanity.com/paper/2206.01349/) (2022)
- [SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer](https://mlsanity.com/paper/2301.12811/) (2023)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [DPImageBench: A Unified Benchmark for Differentially Private Image Synthesis](https://mlsanity.com/paper/2503.14681/) (2025)
- [The Geometric Occam's Razor Implicit in Deep Learning](https://mlsanity.com/paper/2111.15090/) (2021)
- [COIN: Counterfactual Image Generation for VQA Interpretation](https://mlsanity.com/paper/2201.03342/) (2022)
- [Community Research Earth Digital Intelligence Twin (CREDIT)](https://mlsanity.com/paper/2411.07814/) (2024)
- [Physics-assisted Generative Adversarial Network for X-Ray Tomography](https://mlsanity.com/paper/2204.03703/) (2022)
- [Batch Normalization Is Blind to the First and Second Derivatives of the Loss](https://mlsanity.com/paper/2205.15146/) (2022)
- [CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization](https://mlsanity.com/paper/2404.00521/) (2024)
- [Automatic Optimisation of Normalised Neural Networks](https://mlsanity.com/paper/2312.10672/) (2023)
- [Unpaired Image-to-Image Translation for Segmentation and Signal Unmixing](https://mlsanity.com/paper/2505.20746/) (2025)
- [Latent Zoning Network: A Unified Principle for Generative Modeling, Representation Learning, and Classification](https://mlsanity.com/paper/2509.15591/) (2025)
- [Improving GANs with a Feature Cycling Generator](https://mlsanity.com/paper/2210.09638/) (2022)
- [Physics-informed radial basis network (PIRBN): A local approximating neural network for solving nonlinear PDEs](https://mlsanity.com/paper/2304.06234/) (2023)
- [LRM-1B: Towards Large Routing Model](https://mlsanity.com/paper/2507.03300/) (2025)
- [A Reinforcement Learning Approach to Synthetic Data Generation](https://mlsanity.com/paper/2512.21395/) (2025)
- [From Easy to Hard++: Promoting Differentially Private Image Synthesis Through Spatial-Frequency Curriculum](https://mlsanity.com/paper/2601.06368/) (2026)

## Abstract (arXiv)

Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, there is currently limited understanding of why SN is effective. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN for GANs requires no hyper-parameter tuning, whereas stricter implementations of SN have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization and Kaiming initialization. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.