# Collapse of Deep and Narrow Neural Nets

arXiv: https://arxiv.org/abs/1808.04947 · HTML: https://mlsanity.com/paper/1808.04947/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2018.

## Built on by (most cited first, 30 of 37)

- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers](https://mlsanity.com/paper/2504.02508/) (2025)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions](https://mlsanity.com/paper/2102.09924/) (2021)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning](https://mlsanity.com/paper/2505.24061/) (2025)
- [A Hybrid Model and Learning-Based Adaptive Navigation Filter](https://mlsanity.com/paper/2207.12082/) (2022)
- [CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality](https://mlsanity.com/paper/2211.06625/) (2022)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [Estimating permeability of 3D micro-CT images by physics-informed CNNs based on DNS](https://mlsanity.com/paper/2109.01818/) (2021)
- [Combining data assimilation and machine learning to estimate parameters of a convective-scale model](https://mlsanity.com/paper/2109.02953/) (2021)
- [Deep neural networks with controlled variable selection for the identification of putative causal genetic variants](https://mlsanity.com/paper/2109.14719/) (2021)
- [DECONET: an Unfolding Network for Analysis-based Compressed Sensing with Generalization Error Bounds](https://mlsanity.com/paper/2205.07050/) (2022)
- [A prediction and behavioural analysis of machine learning methods for modelling travel mode choice](https://mlsanity.com/paper/2301.04404/) (2023)
- [Modeling non-linear Effects with Neural Networks in Relational Event Models](https://mlsanity.com/paper/2312.12357/) (2023)
- [Multi-layer Perceptron Trainability Explained via Variability](https://mlsanity.com/paper/2105.08911/) (2021)
- [Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization](https://mlsanity.com/paper/2109.11045/) (2021)
- [Testing Feedforward Neural Networks Training Programs](https://mlsanity.com/paper/2204.00694/) (2022)
- [Empirical study of the modulus as activation function in computer vision applications](https://mlsanity.com/paper/2301.05993/) (2023)
- [Hysteresis Activation Function for Efficient Inference](https://mlsanity.com/paper/2411.10573/) (2024)
- [A discrete physics-informed training for projection-based reduced order models with neural networks](https://mlsanity.com/paper/2504.13875/) (2025)
- [Data Topology-Dependent Upper Bounds of Neural Network Widths](https://mlsanity.com/paper/2305.16375/) (2023)
- [DREAM: Debugging and Repairing AutoML Pipelines](https://mlsanity.com/paper/2401.00379/) (2023)
- [Phishing Detection in the Gen-AI Era: Quantized LLMs vs Classical Models](https://mlsanity.com/paper/2507.07406/) (2025)
- [LEA-Net: Layer-wise External Attention Network for Efficient Color Anomaly Detection](https://mlsanity.com/paper/2109.05493/) (2021)
- [Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks](https://mlsanity.com/paper/2109.09612/) (2021)
- [EffCNet: An Efficient CondenseNet for Image Classification on NXP BlueBox](https://mlsanity.com/paper/2111.14243/) (2021)
- [Deeper Learning with CoLU Activation](https://mlsanity.com/paper/2112.12078/) (2021)

## Abstract (arXiv)

Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.