mlsanityData to 14 Oct 2026

On the Crucial Role of Initialization for Matrix Factorization

arXiv 2410.18965 ↗First posted 24 Oct 2024cs.LG eess.SP math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 66 of its 85 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

66 of 85 references.

References · 66

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  13. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  14. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  15. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  16. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  17. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  18. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  19. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
  20. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  21. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  22. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 129
  23. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  24. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  25. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  26. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  27. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  28. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  29. 2013Revisiting the Nystrom Method for Improved Large-Scale Machine Learningcited by 77
  30. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  31. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 71
  32. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 71
  33. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  34. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  35. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  36. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  37. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  38. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 45

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  2. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  3. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  4. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  5. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  6. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
  7. 2025On the Benefits of Weight Normalization for Overparameterized Matrix Sensingcited by 3
  8. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  9. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  10. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  11. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  12. 2025k-SVD with Gradient Descentcited by 0
3 more
  1. 2025TuneComp: Joint Fine-tuning and Compression for Large Foundation Modelscited by 0
  2. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  3. 2026Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Initialization and Regularization of Factorized Neural Layers
  2. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  3. 2025Low-rank Momentum Factorization for Memory Efficient Training
  4. 2025Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
  5. 2025Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
  6. 2026Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer
  7. 2025A space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints
  8. 2026Rotated Manifold Optimization for Low-Rank Adaptation