# On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

arXiv: https://arxiv.org/abs/2004.05867 · HTML: https://mlsanity.com/paper/2004.05867/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 21 of 21)

- [Deep Active Learning by Leveraging Training Dynamics](https://mlsanity.com/paper/2110.08611/) (2021)
- [On the Equivalence between Neural Network and Support Vector Machine](https://mlsanity.com/paper/2111.06063/) (2021)
- [Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks](https://mlsanity.com/paper/2112.05611/) (2021)
- [Convergence of Deep ReLU Networks](https://mlsanity.com/paper/2107.12530/) (2021)
- [Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective](https://mlsanity.com/paper/2103.03113/) (2021)
- [Activation function design for deep networks: linearity and effective initialisation](https://mlsanity.com/paper/2105.07741/) (2021)
- [UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models](https://mlsanity.com/paper/2505.20154/) (2025)
- [Deep equilibrium networks are sensitive to initialization statistics](https://mlsanity.com/paper/2207.09432/) (2022)
- [Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths](https://mlsanity.com/paper/2306.01692/) (2023)
- [Feature Learning and Generalization in Deep Networks with Orthogonal Weights](https://mlsanity.com/paper/2310.07765/) (2023)
- [Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples](https://mlsanity.com/paper/2406.03944/) (2024)
- [On the validity of kernel approximations for orthogonally-initialized neural networks](https://mlsanity.com/paper/2104.05878/) (2021)
- [Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis](https://mlsanity.com/paper/2205.05662/) (2022)
- [Distortion-Disentangled Contrastive Learning](https://mlsanity.com/paper/2303.05066/) (2023)
- [Three Mechanisms of Feature Learning in a Linear Network](https://mlsanity.com/paper/2401.07085/) (2024)
- [Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning](https://mlsanity.com/paper/2302.01002/) (2023)
- [On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2409.05349/) (2024)
- [TLNets: Transformation Learning Networks for long-range time-series prediction](https://mlsanity.com/paper/2305.15770/) (2023)
- [Demystify Optimization and Generalization of Over-parameterized PAC-Bayesian Learning](https://mlsanity.com/paper/2202.01958/) (2022)
- [Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers](https://mlsanity.com/paper/2603.11161/) (2026)
- [Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descent](https://mlsanity.com/paper/2405.07619/) (2024)

## Abstract (arXiv)

The prevailing thinking is that orthogonal weights are crucial to enforcing dynamical isometry and speeding up training. The increase in learning speed that results from orthogonal initialization in linear networks has been well-proven. However, while the same is believed to also hold for nonlinear networks when the dynamical isometry condition is satisfied, the training dynamics behind this contention have not been thoroughly explored. In this work, we study the dynamics of ultra-wide networks across a range of architectures, including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel (NTK). Through a series of propositions and lemmas, we prove that two NTKs, one corresponding to Gaussian weights and one to orthogonal weights, are equal when the network width is infinite. Further, during training, the NTK of an orthogonally-initialized infinite-width network should theoretically remain constant. This suggests that the orthogonal initialization cannot speed up training in the NTK (lazy training) regime, contrary to the prevailing thoughts. In order to explore under what circumstances can orthogonality accelerate training, we conduct a thorough empirical investigation outside the NTK regime. We find that when the hyper-parameters are set to achieve a linear regime in nonlinear activation, orthogonal initialization can improve the learning speed with a large learning rate or large depth.