mlsanityData to 14 Oct 2026

On the Initialisation of Wide Low-Rank Feedforward Neural Networks

arXiv 2301.13710 ↗First posted 31 Jan 2023stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed. Formulae for the optimal weight and bias variances are extended from the full-rank to low-rank setting and are shown to follow from multiplicative scaling. The principle second order effect, the variance of the input-output Jacobian, is derived and shown to increase as the rank to width ratio decreases. These results inform practitioners how to randomly initialize feedforward networks with a reduced number of learnable parameters while in the same ambient dimension, allowing reductions in the computational cost and memory constraints of the associated network.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

13 of 17 references.

References · 13

  1. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  2. 2016Deep Information Propagationcited by 195
  3. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  4. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  5. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  6. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  7. 2019A Mean Field Theory of Batch Normalizationcited by 58
  8. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
  9. 2020Robust Pruning at Initializationcited by 31
  10. 2019Dynamical Isometry and a Mean Field Theory of LSTMs and GRUscited by 17
  11. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  12. 2019Mean field theory for deep dropout networks: digging up gradient backpropagation deeplycited by 4
1 more
  1. 2022Improved Projection Learning for Lower Dimensional Feature Mapscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  2. 2024Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Modelscited by 3
  3. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsitycited by 0
  4. 2026Pruning Deep Neural Networks via the Marchenko--Pastur Distributioncited by 0
  5. 2026Randomly initialized autoencoders: fixed points and edge-of-chaoscited by 0