On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
Not on the map. Why?
Abstract (arXiv)
Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 27 Oct 2022.
Builds on
12 of 25 references.
References · 12
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2014The Loss Surfaces of Multilayer Networkscited by 278
- 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
- 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
- 2016Towards an integration of deep learning and neurosciencecited by 77
- 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
- 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
- 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
- 2016Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausiblecited by 7
- 2017Training Feedforward Neural Networks with Standard Logistic Activations is Feasiblecited by 2
- 2018Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.