Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks
Not on the map. Why?
Abstract (arXiv)
Enforcing orthonormal or isometric property for the weight matrices has been shown to enhance the training of deep neural networks by mitigating gradient exploding/vanishing and increasing the robustness of the learned networks. However, despite its practical performance, the theoretical analysis of orthonormality in neural networks is still lacking; for example, how orthonormality affects the convergence of the training process. In this letter, we aim to bridge this gap by providing convergence analysis for training orthonormal deep linear neural networks. Specifically, we show that Riemannian gradient descent with an appropriate initialization converges at a linear rate for training orthonormal deep linear neural networks with a class of loss functions. Unlike existing works that enforce orthonormal weight matrices for all the layers, our approach excludes this requirement for one layer, which is crucial to establish the convergence guarantee. Our results shed light on how increasing the number of hidden layers can impact the convergence speed. Experimental results validate our theoretical analysis.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 24 Nov 2023.
Builds on
32 of 37 references.
References · 32
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
- 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
- 2015All you need is a good initcited by 124
- 2019Orthogonal Convolutional Neural Networkscited by 122
- 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
- 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
- 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
20 more
- 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
- 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
- 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
- 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
- 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
- 2019Orthogonal Deep Neural Networkscited by 72
- 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
- 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
- 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
- 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
- 2020An Optimal Statistical and Computational Framework for Generalized Tensor Estimationcited by 39
- 2020Controllable Orthogonalization in Training DNNscited by 30
- 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
- 2017The Global Optimization Geometry of Low-Rank Matrix Optimizationcited by 24
- 2022Convergence of gradient descent for deep neural networkscited by 24
- 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
- 2016Generalized BackPropagation, 'Etude De Cas: Orthogonalitycited by 17
- 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
- 2018The Global Optimization Geometry of Shallow Linear Neural Networkscited by 11
- 2016DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagationcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.