mlsanityData to 14 Oct 2026

RaSA: Rank-Sharing Low-Rank Adaptation

arXiv 2503.12576 ↗First posted 16 Mar 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at: https://github.com/zwhe99/RaSA.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

30 of 35 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
18 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  6. 2024LoRA Learns Less and Forgets Lesscited by 328
  7. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  8. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  9. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  10. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  11. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  12. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  13. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  14. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  15. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 29
  16. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  17. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  18. 2024PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 10
  2. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  3. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  4. 2026PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspacescited by 0
  5. 2026Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformerscited by 0
  6. 2026What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BeamLoRA: Beam-Constraint Low-Rank Adaptation
  2. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models
  3. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptation
  4. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  5. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
  6. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  7. 2025LoRA Is Slower Than You Think
  8. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape