mlsanityData to 7 Oct 2026

A Spectral Condition for Feature Learning

arXiv 2310.17813 ↗First posted 26 Oct 2023cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

The push to train ever larger neural networks has motivated the study of initialization and training at large network width. A key challenge is to scale training so that a network's internal representations evolve nontrivially at all widths, a process known as feature learning. Here, we show that feature learning is achieved by scaling the spectral norm of weight matrices and their updates like √textttfan-out/textttfan-in, in contrast to widely used but heuristic scalings based on Frobenius norm and entry size. Our spectral scaling analysis also leads to an elementary derivation of maximal update parametrization. All in all, we aim to provide the reader with a solid conceptual understanding of feature learning in neural networks.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

35 of 50 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
23 more
  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  3. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  6. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  7. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  8. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  9. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  10. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  11. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  12. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  13. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  14. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  15. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  16. 2020On the distance between two neural networks and the stability of learningcited by 46
  17. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  18. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  19. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  20. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  21. 2020Learning compositional functions via multiplicative weight updatescited by 20
  22. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  23. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 34; 2026: 48.

101 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  3. 2024Old Optimizer, New Norm: An Anthologycited by 173
  4. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  5. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  6. 2024Modular Duality in Deep Learningcited by 71
  7. 2024Scalable Optimization in the Modular Normcited by 64
  8. 2025Dion: Distributed Orthonormalized Updatescited by 64
  9. 2025Distillation Scaling Lawscited by 59
  10. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  11. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  12. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
38 more
  1. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  2. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  3. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  4. 2026Controlled LLM Training on Spectral Spherecited by 28
  5. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  6. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  7. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  8. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  9. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  10. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  11. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  12. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  13. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  14. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  15. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  16. 2024Plastic Learning with Deep Fourier Featurescited by 14
  17. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  18. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  19. 2026Replaying pre-training data improves fine-tuningcited by 12
  20. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  21. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  22. 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 11
  23. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  24. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  25. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 10
  26. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  27. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 8
  28. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  29. 2025Beyond Outliers: A Study of Optimizers Under Quantizationcited by 6
  30. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  31. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  32. 2026Muown: Row-Norm Control for Muon Optimizationcited by 6
  33. 2025μnit Scaling: Simple and Scalable FP8 LLM Trainingcited by 5
  34. 2025Optimal Scaling Needs Optimal Normcited by 5
  35. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 5
  36. 2026LpWM: A Case for Sparse Representations in World Modelscited by 5
  37. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  38. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 4

50 most cited of 101 All 101 (JSON)