# Gradient Normalization for Generative Adversarial Networks

arXiv: https://arxiv.org/abs/2109.02235 · HTML: https://mlsanity.com/paper/2109.02235/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Consistency Regularization for Generative Adversarial Networks](https://mlsanity.com/paper/1910.12027/) (2019)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [AutoGAN: Neural Architecture Search for Generative Adversarial Networks](https://mlsanity.com/paper/1908.03835/) (2019)
- [Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect](https://mlsanity.com/paper/1803.01541/) (2018)
- [Spectral Regularization for Combating Mode Collapse in GANs](https://mlsanity.com/paper/1908.10999/) (2019)
- [Adversarial Lipschitz Regularization](https://mlsanity.com/paper/1907.05681/) (2019)
- [Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search](https://mlsanity.com/paper/2007.09180/) (2020)
- [Wasserstein Divergence for GANs](https://mlsanity.com/paper/1712.01026/) (2017)
- [On the Effects of Batch and Weight Normalization in Generative Adversarial Networks](https://mlsanity.com/paper/1704.03971/) (2017)

## Built on by (most cited first, 25 of 25)

- [Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint](https://mlsanity.com/paper/2305.13599/) (2023)
- [Robust low-rank training via approximate orthonormal constraints](https://mlsanity.com/paper/2306.01485/) (2023)
- [SCAM! Transferring humans between images with Semantic Cross Attention Modulation](https://mlsanity.com/paper/2210.04883/) (2022)
- [Balanced Training for Sparse GANs](https://mlsanity.com/paper/2302.14670/) (2023)
- [Generative Convolution Layer for Image Generation](https://mlsanity.com/paper/2111.15171/) (2021)
- [Scalable GANs with Transformers](https://mlsanity.com/paper/2509.24935/) (2025)
- [A Novel Generator with Auxiliary Branch for Improving GAN Performance](https://mlsanity.com/paper/2112.14968/) (2021)
- [BMD-GAN: Bone mineral density estimation using x-ray image decomposition into projections of bone-segmented quantitative computed tomography using hierarchical learning](https://mlsanity.com/paper/2207.03210/) (2022)
- [GNPassGAN: Improved Generative Adversarial Networks For Trawling Offline Password Guessing](https://mlsanity.com/paper/2208.06943/) (2022)
- [Bone mineral density estimation from a plain X-ray image by learning decomposition into projections of bone-segmented computed tomography](https://mlsanity.com/paper/2307.11513/) (2023)
- [Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games](https://mlsanity.com/paper/2505.19537/) (2025)
- [Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation](https://mlsanity.com/paper/2210.10469/) (2022)
- [Leveraging Contaminated Datasets to Learn Clean-Data Distribution with Purified Generative Adversarial Networks](https://mlsanity.com/paper/2302.01722/) (2023)
- [Structural Adversarial Objectives for Self-Supervised Representation Learning](https://mlsanity.com/paper/2310.00357/) (2023)
- [GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/2111.03162/) (2021)
- [On Deep Learning in Password Guessing, a Survey](https://mlsanity.com/paper/2208.10413/) (2022)
- [Neural Collage Transfer: Artistic Reconstruction via Material Manipulation](https://mlsanity.com/paper/2311.02202/) (2023)
- [A Comprehensive Survey on Synthetic Infrared Image synthesis](https://mlsanity.com/paper/2408.06868/) (2024)
- [Diversifying Long Prompt Image Generation through Structured Prompt Embedding Space Sampling](https://mlsanity.com/paper/2511.20251/) (2025)
- [Effective Shortcut Technique for GAN](https://mlsanity.com/paper/2201.11351/) (2022)
- [Improving GANs with a Feature Cycling Generator](https://mlsanity.com/paper/2210.09638/) (2022)
- [Penalty Gradient Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/2306.13576/) (2023)
- [Stability Analysis Framework for Particle-based Distance GANs with Wasserstein Gradient Flow](https://mlsanity.com/paper/2307.01879/) (2023)
- [EqGAN: Feature Equalization Fusion for Few-shot Image Generation](https://mlsanity.com/paper/2307.14638/) (2023)
- [Accelerated Neural Network Training with Rooted Logistic Objectives](https://mlsanity.com/paper/2310.03890/) (2023)

## Abstract (arXiv)

In this paper, we propose a novel normalization method called gradient normalization (GN) to tackle the training instability of Generative Adversarial Networks (GANs) caused by the sharp gradient space. Unlike existing work such as gradient penalty and spectral normalization, the proposed GN only imposes a hard 1-Lipschitz constraint on the discriminator function, which increases the capacity of the discriminator. Moreover, the proposed gradient normalization can be applied to different GAN architectures with little modification. Extensive experiments on four datasets show that GANs trained with gradient normalization outperform existing methods in terms of both Frechet Inception Distance and Inception Score.