# Deep orthogonal linear networks are shallow

arXiv: https://arxiv.org/abs/2011.13831 · HTML: https://mlsanity.com/paper/2011.13831/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 1 of 1)

- [On the Role of Initialization on the Implicit Bias in Deep Linear Networks](https://mlsanity.com/paper/2402.02454/) (2024)

## Abstract (arXiv)

We consider the problem of training a deep orthogonal linear network, which consists of a product of orthogonal matrices, with no non-linearity in-between. We show that training the weights with Riemannian gradient descent is equivalent to training the whole factorization by gradient descent. This means that there is no effect of overparametrization and implicit bias at all in this setting: training such a deep, overparametrized, network is perfectly equivalent to training a one-layer shallow network.