# The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

arXiv: https://arxiv.org/abs/1712.05577 · HTML: https://mlsanity.com/paper/1712.05577/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2017.

## Built on by (most cited first, 14 of 14)

- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [A Kernel Perspective of Skip Connections in Convolutional Networks](https://mlsanity.com/paper/2211.14810/) (2022)
- [Optimized and autonomous machine learning framework for characterizing pores, particles, grains and grain boundaries in microstructural images](https://mlsanity.com/paper/2101.06474/) (2021)
- [Self-Supervised Learning of Linear Precoders under Non-Linear PA Distortion for Energy-Efficient Massive MIMO Systems](https://mlsanity.com/paper/2210.07037/) (2022)
- [ANAct: Adaptive Normalization for Activation Functions](https://mlsanity.com/paper/2208.13315/) (2022)
- [Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equations](https://mlsanity.com/paper/2407.00568/) (2024)
- [Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters](https://mlsanity.com/paper/2410.01239/) (2024)
- [Detecting AI Assistance in Abstract Complex Tasks](https://mlsanity.com/paper/2507.10761/) (2025)
- [Abnormal Occupancy Grid Map Recognition using Attention Network](https://mlsanity.com/paper/2110.09047/) (2021)
- [Connecting the Dots: Graph Neural Network Powered Ensemble and Classification of Medical Images](https://mlsanity.com/paper/2311.07321/) (2023)
- [The Heat is On: Thermal Facial Landmark Tracking](https://mlsanity.com/paper/2311.08308/) (2023)
- [GRN+: A Simplified Generative Reinforcement Network for Tissue Layer Analysis in 3D Ultrasound Images for Chronic Low-back Pain](https://mlsanity.com/paper/2503.19736/) (2025)

## Abstract (arXiv)

Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the *collapsing domain problem*, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that *any neural network is a residual network*, we devise the *residual trick*, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.