mlsanityData to 7 Oct 2026

The Emergence of Spectral Universality in Deep Networks

arXiv 1802.09979 ↗First posted 27 Feb 2018stat.ML cs.LGcited by 74

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 17; 2023: 14; 2024: 10; 2025: 7; 2026: 9.

74 papers.

Sort:

By area:

  1. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  2. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  3. 2021Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifierscited by 56
  4. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  5. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  6. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  7. 2023Deep networks for system identification: a Surveycited by 35
  8. 2024Learning Continually by Spectral Regularizationcited by 34
  9. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  10. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  11. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  12. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
38 more
  1. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  2. 2022Random matrix analysis of deep neural network weight matricescited by 21
  3. 2022Overparameterized Linear Regression under Adversarial Attackscited by 21
  4. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  5. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  6. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  7. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  8. 2022On the Learning and Learnability of Quasimetricscited by 15
  9. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  10. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  11. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  12. 2023Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspectivecited by 10
  13. 2023Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probabilitycited by 10
  14. 2022Fractional SDE-Net: Generation of Time Series Data with Long-term Memorycited by 9
  15. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  16. 2022C^*-algebra Net: A New Approach Generalizing Neural Network Parameters to C^*-algebracited by 9
  17. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
  18. 2024Unitary convolutions for learning on graphs and groupscited by 8
  19. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  20. 2021Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Casecited by 5
  21. 2023On the Initialisation of Wide Low-Rank Feedforward Neural Networkscited by 5
  22. 2023Feature Learning and Generalization in Deep Networks with Orthogonal Weightscited by 5
  23. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  24. 2024Deep Neural Network Initialization with Sparsity Inducing Activationscited by 5
  25. 2021Better Training using Weight-Constrained Stochastic Dynamicscited by 4
  26. 2021Lottery Tickets with Nonzero Biasescited by 4
  27. 2022Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Casecited by 4
  28. 2022Universal characteristics of deep neural network loss surfaces from random matrix theorycited by 4
  29. 2022Optimal Activation Functions for the Random Features Regression Modelcited by 4
  30. 2021Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networkscited by 3
  31. 2021Free Probability for predicting the performance of feed-forward fully connected neural networkscited by 3
  32. 2022AutoInit: Automatic Initialization via Jacobian Tuningcited by 3
  33. 2023Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamicscited by 3
  34. 2024Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networkscited by 3
  35. 2024Sharper Bounds for Chebyshev Moment Matching, with Applicationscited by 3
  36. 2021A Weight Initialization Based on the Linear Product Structure for Neural Networkscited by 2
  37. 2022On Bridging the Gap between Mean Field and Finite Width in Deep Random Neural Networks with Batch Normalizationcited by 2
  38. 2022Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channelscited by 2

50 most cited of 74 All 74 (JSON)