mlsanityData to 7 Oct 2026

On the validity of kernel approximations for orthogonally-initialized neural networks

arXiv 2104.05878 ↗First posted 13 Apr 2021cs.LG cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

In this note we extend kernel function approximation results for neural networks with Gaussian-distributed weights to single-layer networks initialized using Haar-distributed random orthogonal matrices (with possible rescaling). This is accomplished using recent results from random matrix theory.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 10 of its 15 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

10 of 15 references.

References · 10

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  5. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  6. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  7. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  8. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  9. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  10. 2018Information Geometry of Orthogonal Initializations and Trainingcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1.

4 papers.

  1. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  2. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  3. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  4. 2022Deep equilibrium networks are sensitive to initialization statisticscited by 8