mlsanityData to 7 Oct 2026

Tensor Programs III: Neural Matrix Laws

arXiv 2009.10685 ↗First posted 22 Sep 2020cs.NE math.PRcited by 47

Not on the map. Why?

Abstract (arXiv)

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 4; 2024: 3; 2025: 11; 2026: 10.

47 papers.

Sort:

By area:

  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  2. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  3. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  4. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  5. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  6. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  7. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  8. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  9. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  10. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  11. 2022Neural Tangent Kernel: A Surveycited by 22
  12. 2022On the infinite-depth limit of finite-width neural networkscited by 22
35 more
  1. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  2. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  3. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  4. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  5. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  6. 2022A duality connecting neural network and cosmological dynamicscited by 10
  7. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  8. 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
  9. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  10. 2021Regularization in ResNet with Stochastic Depthcited by 7
  11. 2021Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limitcited by 7
  12. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  13. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  14. 2021Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Casecited by 5
  15. 2025Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscited by 5
  16. 2022Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Casecited by 4
  17. 2026JPmHC Dynamical Isometry via Orthogonal Hyper-Connectionscited by 4
  18. 2025Function-Space Learning Ratescited by 3
  19. 2025Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrizationcited by 3
  20. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 3
  21. 2021Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottleneckscited by 2
  22. 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
  23. 2022Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networkscited by 1
  24. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  25. 2025Random weights of DNNs and emergence of fixed pointscited by 1
  26. 2026LoRDO: Distributed Low-Rank Optimization with Infrequent Communicationcited by 1
  27. 2026Learning Rate Transfer in Normalized Transformerscited by 1
  28. 2026GQA-μP: The maximal parameterization update for grouped query attentioncited by 1
  29. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  30. 2025Feature Learning Dynamics in Infinite-Depth Neural Networkscited by 0
  31. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descentcited by 0
  32. 2026How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrencescited by 0
  33. 2026Canonical Regularisation of Wide Feature-Learning Neural Networkscited by 0
  34. 2026Quantitative Gaussian-Process limits of Tensor Programscited by 0
  35. 2026σTransfer: Uncertainty Transfer from Small to Large Networks under μPcited by 0