mlsanityData to 7 Oct 2026

The Impact of Initialization on LoRA Finetuning Dynamics

arXiv 2406.08447 ↗First posted 12 Jun 2024cs.LG cs.AI cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021). Essentially, to start from the pretrained model as initialization for finetuning, one can either initialize B to zero and A to random (default initialization in PEFT package), or vice-versa. In both cases, the product BA is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly similar. They should in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an incorrect intuition and that the first scheme (initializing B to zero and A to random) on average yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initialization, resulting in more efficient learning of the first scheme. We validate our results with extensive experiments on LLMs.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

44 of 51 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
32 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  10. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  11. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  12. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  13. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  14. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  15. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  16. 2016Deep Information Propagationcited by 194
  17. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  18. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  19. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  20. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  21. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  22. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  23. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  24. 2020Stable ResNetcited by 49
  25. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  26. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  27. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  28. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  29. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 35
  30. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  31. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  32. 2023Width and Depth Limits Commute in Residual Networkscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 16.

53 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  3. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  4. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  6. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  7. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  8. 2025Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Modelscited by 10
  9. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  10. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  11. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  12. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
38 more
  1. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  2. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 8
  3. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  4. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  5. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  6. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5
  7. 2024Parameter-Efficient Fine-Tuning via Circular Convolutioncited by 4
  8. 2025Curvature Tuning: Provable Training-free Model Steering From a Single Parametercited by 4
  9. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  10. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4
  11. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  12. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  13. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4
  14. 2025Optimization-Inspired Few-Shot Adaptation for Large Language Modelscited by 3
  15. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  16. 2025Low-Rank Adaptation of Neural Fieldscited by 2
  17. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  18. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  19. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  20. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  21. 2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2
  22. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  23. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  24. 2025Norm-Bounded Low-Rank Adaptationcited by 0
  25. 2025D2LoRA: Data-Driven LoRA Initialization for Low Resource Taskscited by 0
  26. 2025A Physics-Inspired Optimizer: Velocity Regularized Adamcited by 0
  27. 2025TuneComp: Joint Fine-tuning and Compression for Large Foundation Modelscited by 0
  28. 2025Reimagining Parameter Space Exploration with Diffusion Modelscited by 0
  29. 2025IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptationcited by 0
  30. 2025HyperAdapt: Simple High-Rank Adaptationcited by 0
  31. 2025α-LoRA: Effective Fine-Tuning via Base Model Rescalingcited by 0
  32. 2025LoRA-Edge: Tensor-Train-Assisted LoRA for Practical CNN Fine-Tuning on Edge Devicescited by 0
  33. 2026An Ubuntu-Guided Large Language Model Framework for Cognitive Behavioral Mental Health Dialoguecited by 0
  34. 2026Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptationcited by 0
  35. 2026WaDi: Weight Direction-aware Distillation for One-step Image Synthesiscited by 0
  36. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0
  37. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  38. 2026Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuningcited by 0

50 most cited of 53 All 53 (JSON)