Data to 7 Oct 2026

Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?

arXiv 1801.03744 ↗First posted 11 Jan 2018stat.ML cs.LG math.PRcited by 94

Not on the map. Why?

Abstract (arXiv)

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, given by the sum of the reciprocals of the hidden layer widths. When beta is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 11; 2023: 27; 2024: 18; 2025: 14; 2026: 7.

94 papers.

Sort:

By area:

2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
2022Accelerating Dataset Distillation via Model Augmentationcited by 62
2021Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifierscited by 56
2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
2023Hidden symmetries of ReLU networkscited by 38
2023Deep networks for system identification: a Surveycited by 35
2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 33
2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30

38 more

2023A Survey on Reservoir Computing and its Interdisciplinary Applications Beyond Traditional Machine Learningcited by 22
2022Bayesian Interpolation with Deep Linear Networkscited by 21
2024Parseval Regularization for Continual Reinforcement Learningcited by 21
2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
2023Quantitative CLTs in Deep Neural Networkscited by 19
2021Deep ReLU Networks Preserve Expected Lengthcited by 17
2023The Cascaded Forward Algorithm for Neural Network Trainingcited by 17
2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
2021Activation function design for deep networks: linearity and effective initialisationcited by 13
2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
2023CS4ML: A general framework for active learning with arbitrary data based on Christoffel functionscited by 11
2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
2023Depth Dependence of μP Learning Rates in ReLU MLPscited by 9
2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
2024On the Vulnerability of Skip Connections to Model Inversion Attackscited by 8
2022Graph Neural Networks Extract High-Resolution Cultivated Land Maps from Sentinel-2 Image Seriescited by 6
2022CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learningcited by 6
2023Field theory for optimal signal propagation in ResNetscited by 6
2024Accurate Explanation Model for Image Classifiers using Class Association Embeddingcited by 6
2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
2021Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networkscited by 5
2021AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networkscited by 5
2023Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performancecited by 5
2023Feature Learning and Generalization in Deep Networks with Orthogonal Weightscited by 5
2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
2021Deep Neural Networks to Recover Unknown Physical Parameters from Oscillating Time Seriescited by 4
2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
2024Cross-domain Learning Framework for Tracking Users in RIS-aided Multi-band ISAC Systems with Sparse Labeled Datacited by 4
2022On Scrambling Phenomena for Randomly Initialized Recurrent Networkscited by 3
2023Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initializationcited by 3
2023Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activationscited by 3
2023The Evolution of the Interplay Between Input Distributions and Linear Regions in Networkscited by 3
2024Improved deep learning of chaotic dynamical systems with multistep penalty lossescited by 3
2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3

50 most cited of 94 All 94 (JSON)