mlsanityData to 7 Oct 2026

Low Rank Gradients and Where to Find Them

arXiv 2510.01303 ↗First posted 1 Oct 2025cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper investigates low-rank structure in the gradients of the training loss for two-layer neural networks while relaxing the usual isotropy assumptions on the training data and parameters. We consider a spiked data model in which the bulk can be anisotropic and ill-conditioned, we do not require independent data and weight matrices and we also analyze both the mean-field and neural-tangent-kernel scalings. We show that the gradient with respect to the input weights is approximately low rank and is dominated by two rank-one terms: one aligned with the bulk data-residue , and another aligned with the rank one spike in the input data. We characterize how properties of the training data, the scaling regime and the activation function govern the balance between these two components. Additionally, we also demonstrate that standard regularizers, such as weight decay, input noise and Jacobian penalties, also selectively modulate these components. Experiments on synthetic and real data corroborate our theoretical predictions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

34 of 47 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  12. 2018Gradient descent aligns the layers of deep linear networkscited by 182
22 more
  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  4. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  5. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  6. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  7. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  8. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  9. 2022Dimension free ridge regressioncited by 50
  10. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  11. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  12. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  13. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  14. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  15. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  16. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  17. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  18. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  19. 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 13
  20. 2021Analysis of One-Hidden-Layer Neural Networks via the Resolvent Methodcited by 9
  21. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  22. 2023Least Squares Regression Can Exhibit Under-Parameterized Double Descentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  2. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0
  3. 2026Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learningcited by 0
  4. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0