mlsanityData to 14 Oct 2026

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

arXiv 2505.18877 ↗First posted 24 May 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

54 of 70 references.

References · 54

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  4. 2021Convex optimizationcited by 2,515
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  13. 2022Galactica: A Large Language Model for Sciencecited by 744
  14. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  15. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  16. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  17. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  18. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  19. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  20. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  21. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  22. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  23. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  24. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  25. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  26. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  27. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  28. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  29. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  30. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  31. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  32. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 71
  33. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 71
  34. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  35. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  36. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  37. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  38. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  2. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
  3. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  4. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  5. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuningcited by 1
  6. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 1
  7. 2026One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuningcited by 0
  8. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  9. 2026What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models
  2. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  3. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  4. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  6. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  7. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  8. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape