mlsanityData to 14 Oct 2026

Collapse of Deep and Narrow Neural Nets

arXiv 1808.04947 ↗First posted 15 Aug 2018stat.ML cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 15; 2022: 7; 2023: 7; 2024: 1; 2025: 7.

37 papers.

Sort:

By area:

  1. 2021On the approximation of functions by tanh neural networkscited by 81
  2. 2025APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformerscited by 19
  3. 2021Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domainscited by 18
  4. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  5. 2021Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Detailscited by 14
  6. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  7. 2022A Hybrid Model and Learning-Based Adaptive Navigation Filtercited by 12
  8. 2022CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimalitycited by 12
  9. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  10. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  11. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  12. 2021Estimating permeability of 3D micro-CT images by physics-informed CNNs based on DNScited by 5
25 more
  1. 2021Combining data assimilation and machine learning to estimate parameters of a convective-scale modelcited by 4
  2. 2021Deep neural networks with controlled variable selection for the identification of putative causal genetic variantscited by 4
  3. 2022DECONET: an Unfolding Network for Analysis-based Compressed Sensing with Generalization Error Boundscited by 4
  4. 2023A prediction and behavioural analysis of machine learning methods for modelling travel mode choicecited by 4
  5. 2023Modeling non-linear Effects with Neural Networks in Relational Event Modelscited by 3
  6. 2021Multi-layer Perceptron Trainability Explained via Variabilitycited by 2
  7. 2021Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularizationcited by 2
  8. 2022Testing Feedforward Neural Networks Training Programscited by 2
  9. 2023Empirical study of the modulus as activation function in computer vision applicationscited by 2
  10. 2024Hysteresis Activation Function for Efficient Inferencecited by 2
  11. 2025A discrete physics-informed training for projection-based reduced order models with neural networkscited by 2
  12. 2023Data Topology-Dependent Upper Bounds of Neural Network Widthscited by 1
  13. 2023DREAM: Debugging and Repairing AutoML Pipelinescited by 1
  14. 2025Phishing Detection in the Gen-AI Era: Quantized LLMs vs Classical Modelscited by 1
  15. 2021LEA-Net: Layer-wise External Attention Network for Efficient Color Anomaly Detectioncited by 0
  16. 2021Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networkscited by 0
  17. 2021EffCNet: An Efficient CondenseNet for Image Classification on NXP BlueBoxcited by 0
  18. 2021Deeper Learning with CoLU Activationcited by 0
  19. 2022PEA: Improving the Performance of ReLU Networks for Free by Using Progressive Ensemble Activationscited by 0
  20. 2022EraseNet: A Recurrent Residual Network for Supervised Document Cleaningcited by 0
  21. 2022Nish: A Novel Negative Stimulated Hybrid Activation Functioncited by 0
  22. 2023Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functionscited by 0
  23. 2025How to warm-start your unfolding networkcited by 0
  24. 2025An in-depth look at approximation via deep and narrow neural networkscited by 0
  25. 2025SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximationcited by 0