mlsanityData to 7 Oct 2026

Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian

arXiv 1906.05392 ↗First posted 12 Jun 2019cs.LG math.OC stat.MLcited by 67

Not on the map. Why?

Abstract (arXiv)

Modern neural network architectures often generalize well despite containing many more parameters than the size of the training dataset. This paper explores the generalization capabilities of neural networks trained via gradient descent. We develop a data-dependent optimization and generalization theory which leverages the low-rank structure of the Jacobian matrix associated with the network. Our results help demystify why training and generalization is easier on clean and structured datasets and harder on noisy and unstructured datasets as well as how the network size affects the evolution of the train and test errors during training. Specifically, we use a control knob to split the Jacobian spectum into "information" and "nuisance" spaces associated with the large and small singular values. We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well. Over the nuisance space training is slower and early stopping can help with generalization at the expense of some bias. We also show that the overall generalization capability of the network is controlled by how well the label vector is aligned with the information space. A key feature of our results is that even constant width neural nets can provably generalize for sufficiently nice datasets. We conduct various numerical experiments on deep networks that corroborate our theoretical findings and demonstrate that: (i) the Jacobian of typical neural networks exhibit low-rank structure with a few large singular values and many small ones leading to a low-dimensional information space, (ii) over the information space learning is fast and most of the label vector falls on this space, and (iii) label noise falls on the nuisance space and impedes optimization/generalization.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 11; 2023: 7; 2024: 17; 2025: 8; 2026: 9.

67 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  4. 2022Investigating Why Contrastive Learning Benefits Robustness Against Label Noisecited by 46
  5. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  6. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  7. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  8. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  9. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  10. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  11. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  12. 2021Kernelized Heterogeneous Risk Minimizationcited by 21
38 more
  1. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  2. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  3. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  4. 2025LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptationcited by 17
  5. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  6. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  7. 2021Understanding Overparameterization in Generative Adversarial Networkscited by 14
  8. 2021Generalization Guarantees for Neural Architecture Search with Train-Validation Splitcited by 14
  9. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  10. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  11. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  12. 2021Relative stability toward diffeomorphisms indicates performance in deep netscited by 12
  13. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  14. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  15. 2021Provable Super-Convergence with a Large Cyclical Learning Ratecited by 10
  16. 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
  17. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  18. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 9
  19. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  20. 2021DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Modelscited by 8
  21. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  22. 2026On the Non-Identifiability of Steering Vectors in Large Language Modelscited by 8
  23. 2021Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classificationcited by 7
  24. 2024Solving Inverse Problems via Diffusion Optimal Controlcited by 7
  25. 2021Deep Networks Provably Classify Data on Curvescited by 6
  26. 2021Representation Alignment in Neural Networkscited by 6
  27. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  28. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  29. 2021Towards Understanding Generalization via Decomposing Excess Risk Dynamicscited by 5
  30. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  31. 2022Data-Efficient Augmentation for Training Neural Networkscited by 4
  32. 2024Revealing the Utilized Rank of Subspaces of Learning in Neural Networkscited by 4
  33. 2026Weight Decay Improves Language Model Plasticitycited by 4
  34. 2021On the Impact of Stable Ranks in Deep Netscited by 3
  35. 2023Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddingscited by 3
  36. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  37. 2024GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compressioncited by 3
  38. 2026CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editingcited by 3

50 most cited of 67 All 67 (JSON)