# Why ReLU Units Sometimes Die: Analysis of Single-Unit Error Backpropagation in Neural Networks

arXiv: https://arxiv.org/abs/1812.05981 · HTML: https://mlsanity.com/paper/1812.05981/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 12 of 12)

- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks](https://mlsanity.com/paper/2108.12943/) (2021)
- [Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks](https://mlsanity.com/paper/2201.06126/) (2022)
- [NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis](https://mlsanity.com/paper/2409.11836/) (2024)
- [Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life](https://mlsanity.com/paper/2302.05818/) (2023)
- [Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks](https://mlsanity.com/paper/2410.16958/) (2024)
- [Regularization by Misclassification in ReLU Neural Networks](https://mlsanity.com/paper/2111.02154/) (2021)
- [A Diagnostic Evaluation of Neural Networks Trained with the Error Diffusion Learning Algorithm](https://mlsanity.com/paper/2504.14814/) (2025)
- [Learning dynamical systems with biochemically informed neural ordinary differential equations](https://mlsanity.com/paper/2605.24170/) (2026)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Pioneering Precision in Lumbar Spine MRI Segmentation with Advanced Deep Learning and Data Enhancement](https://mlsanity.com/paper/2409.06018/) (2024)
- [Control of dynamical systems with neural networks](https://mlsanity.com/paper/2510.12810/) (2025)

## Abstract (arXiv)

Recently, neural networks in machine learning use rectified linear units (ReLUs) in early processing layers for better performance. Training these structures sometimes results in "dying ReLU units" with near-zero outputs. We first explore this condition via simulation using the CIFAR-10 dataset and variants of two popular convolutive neural network architectures. Our explorations show that the output activation probability Pr[y>0] is generally less than 0.5 at system convergence for layers that do not employ skip connections, and this activation probability tends to decrease as one progresses from input layer to output layer. Employing a simplified model of a single ReLU unit trained by a variant of error backpropagation, we then perform a statistical convergence analysis to explore the model's evolutionary behavior. Our analysis describes the potentially-slower convergence speeds of dying ReLU units, and this issue can occur regardless of how the weights are initialized.