# Using linear initialisation to improve speed of convergence and fully-trained error in Autoencoders

arXiv: https://arxiv.org/abs/2311.10699 · HTML: https://mlsanity.com/paper/2311.10699/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 3 of 3)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Empirical comparison between autoencoders and traditional dimensionality reduction methods](https://mlsanity.com/paper/2103.04874/) (2021)
- [Fair Adversarial Networks](https://mlsanity.com/paper/2002.12144/) (2020)

## Abstract (arXiv)

Good weight initialisation is an important step in successful training of Artificial Neural Networks. Over time a number of improvements have been proposed to this process. In this paper we introduce a novel weight initialisation technique called the Straddled Matrix Initialiser. This initialisation technique is motivated by our assumption that major, global-scale relationships in data are linear with only smaller effects requiring complex non-linearities. Combination of Straddled Matrix and ReLU activation function initialises a Neural Network as a de facto linear model, which we postulate should be a better starting point for optimisation given our assumptions. We test this by training autoencoders on three datasets using Straddled Matrix and seven other state-of-the-art weight initialisation techniques. In all our experiments the Straddeled Matrix Initialiser clearly outperforms all other methods.