How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs
Not on the map. Why?
Abstract (arXiv)
The Edge-of-Chaos (EoC) theory developed for the random initialization of deep networks allows more efficient training by both preserving information in the initial outputs of the network and minimising exploding or vanishing gradients through characterisation of the intermediate layers as Gaussian processes. This EoC theory provides formulae for the choice of the initialisation distribution variances of the weights and biases. For activations which are approximately linear around the origin, the EoC theory typically encourages the Gaussian process variance to converge towards zero with increasing depth. Here we consider the less studied setting of highly sparsity inducing activations where a large region of values near the origin are set to zero. In this setting we prove a new phenomenon whereby initialisations leading to larger fixed Gaussian processes are beneficial to training stability. This theory informs a new, yet simple, initialisation strategy that allows training DNNs and CNNs with as large as 90% sparsity in the hidden layers.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 5 Feb 2026.
Builds on
30 of 41 references.
References · 30
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2020What is the State of Neural Network Pruning?cited by 578
- 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
- 2016Deep Information Propagationcited by 194
- 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
- 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
18 more
- 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
- 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2018The Emergence of Spectral Universality in Deep Networkscited by 74
- 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
- 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
- 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
- 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
- 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
- 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
- 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
- 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
- 2021Neural Network Quantization for Efficient Inference: A Surveycited by 13
- 2023Sparsity-aware generalization theory for deep neural networkscited by 8
- 2024Learning Neural Networks with Sparse Activationscited by 6
- 2024Deep Neural Network Initialization with Sparsity Inducing Activationscited by 5
- 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 5
- 2025Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networkscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.