mlsanityData to 14 Oct 2026

An Overview of Low-Rank Structures in the Training and Adaptation of Large Models

arXiv 2503.19859 ↗First posted 25 Mar 2025cs.LG eess.SP math.OCcited by 9

Not on the map. Why?

Abstract (arXiv)

The substantial computational demands of modern large-scale deep learning present significant challenges for efficient training and deployment. Recent research has revealed a widespread phenomenon wherein deep networks inherently learn low-rank structures in their weights and representations during training. This tutorial paper provides a comprehensive review of advances in identifying and exploiting these low-rank structures, bridging mathematical foundations with practical applications. We present two complementary theoretical perspectives on the emergence of low-rankness: viewing it through the optimization dynamics of gradient descent throughout training, and understanding it as a result of implicit regularization effects at convergence. Practically, these theoretical perspectives provide a foundation for understanding the success of techniques such as Low-Rank Adaptation (LoRA) in fine-tuning, inspire new parameter-efficient low-rank training strategies, and explain the effectiveness of masked training approaches like dropout and masked self-supervised learning.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
29 more
  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  3. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  4. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  6. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  7. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  8. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  9. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  10. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  12. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  13. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 44
  14. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 32
  15. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  16. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  17. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  18. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  19. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  20. 2018On the Implicit Bias of Dropoutcited by 22
  21. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  22. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 20
  23. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 17
  24. 2017Dropout as a Low-Rank Regularizer for Matrix Factorizationcited by 14
  25. 2019On Dropout and Nuclear Norm Regularizationcited by 12
  26. 2024Residual Connections Harm Generative Representation Learningcited by 11
  27. 2023Representation Deficiency in Masked Language Modelingcited by 10
  28. 2019On the Regularization Properties of Structured Dropoutcited by 5
  29. 2024FastLRNR and Sparse Physics Informed Backpropagationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
  2. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  3. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  4. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 6
  5. 2026Pointwise Generalization in Deep Neural Networkscited by 4
  6. 2026Weight space Detection of Backdoors in LoRA Adapterscited by 1
  7. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  8. 2026Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMscited by 0
  9. 2026DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNscited by 0