mlsanityData to 7 Oct 2026

A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

arXiv 2312.03732 ↗First posted 28 Nov 2023cs.CL cs.LGcited by 231

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) have become increasingly compute and memory intensive, parameter-efficient fine-tuning (PEFT) methods are now a common strategy to fine-tune LLMs. A popular PEFT method is Low-Rank Adapters (LoRA), which adds trainable low-rank "adapters" to selected layers. Each adapter consists of a low-rank matrix product, multiplicatively scaled by a rank-dependent factor. This scaling factor, which divides adapters by a factor of the rank, results in slowed learning and stunted performance for LoRA with higher-rank adapters. Consequently, the use of LoRA in practice has generally been limited to very low ranks. In this work, we study the impact of the scaling factor on the learning process and prove that LoRA adapters should be divided by a factor of the square root of the rank. Modifying LoRA with the appropriate scaling factor, which we call the rank-stabilized LoRA (rsLoRA) method, easily provides for a fine-tuning compute/performance trade-off, where larger ranks can be used to trade off increased computational resources during training for better fine-tuning performance, with no change in inference computing cost.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Builds on 13 of its 19 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

13 of 19 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  11. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  12. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
1 more
  1. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 94; 2026: 96.

231 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  3. 2024LoRA Learns Less and Forgets Lesscited by 313
  4. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  5. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  6. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  7. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  8. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  9. 2025Model Organisms for Emergent Misalignmentcited by 84
  10. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  11. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  12. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
38 more
  1. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  2. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  3. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  4. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 36
  5. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  6. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 34
  7. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  8. 2024FRoundation: Are Foundation Models Ready for Face Recognition?cited by 22
  9. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  10. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 19
  11. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
  12. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  13. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  14. 2025Instruction-Tuning LLMs for Event Extraction with Annotation Guidelinescited by 18
  15. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  16. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 18
  17. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  18. 2024Sparse High Rank Adapterscited by 17
  19. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 17
  20. 2024FouRA: Fourier Low Rank Adaptationcited by 15
  21. 2025FoundPAD: Foundation Models Reloaded for Face Presentation Attack Detectioncited by 14
  22. 2025MADation: Face Morphing Attack Detection with Foundation Modelscited by 14
  23. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
  24. 2026Simulated Students in Tutoring Dialogues: Substance or Illusion?cited by 14
  25. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  26. 2024LangGFM: A Large Language Model Alone Can be a Powerful Graph Foundation Modelcited by 13
  27. 2025How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularizationcited by 13
  28. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  29. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  30. 2025Starjob: Dataset for LLM-Driven Job Shop Schedulingcited by 12
  31. 2025Evaluating Parameter Efficient Methods for RLVRcited by 12
  32. 2026ALL-FEM: Agentic Large Language models Fine-tuned for Finite Element Methodscited by 12
  33. 2024Co-occurrence is not Factual Association in Language Modelscited by 11
  34. 2024Language Models are Symbolic Learners in Arithmeticcited by 11
  35. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  36. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 11
  37. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  38. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 10

50 most cited of 231 All 231 (JSON)