# On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

arXiv: https://arxiv.org/abs/2211.08408 · HTML: https://mlsanity.com/paper/2211.08408/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?](https://mlsanity.com/paper/1810.09102/) (2018)
- [All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation](https://mlsanity.com/paper/1703.01827/) (2017)
- [Recurrent Orthogonal Networks and Long-Memory Tasks](https://mlsanity.com/paper/1602.06662/) (2016)
- [Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible](https://mlsanity.com/paper/1606.01651/) (2016)
- [Training Feedforward Neural Networks with Standard Logistic Activations is Feasible](https://mlsanity.com/paper/1710.01013/) (2017)
- [Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learning](https://mlsanity.com/paper/1803.09119/) (2018)

## Built on by (most cited first, 1 of 1)

- [Can Local Representation Alignment RNNs Solve Temporal Tasks?](https://mlsanity.com/paper/2504.13531/) (2025)

## Abstract (arXiv)

Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.