# RaSA: Rank-Sharing Low-Rank Adaptation

arXiv: https://arxiv.org/abs/2503.12576 · HTML: https://mlsanity.com/paper/2503.12576/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models](https://mlsanity.com/paper/2309.11674/) (2023)
- [Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying](https://mlsanity.com/paper/2311.09578/) (2023)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.12130/) (2024)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead](https://mlsanity.com/paper/2407.00066/) (2024)
- [ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation](https://mlsanity.com/paper/2406.10785/) (2024)
- [PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation](https://mlsanity.com/paper/2401.11316/) (2024)

## Built on by (most cited first, 6 of 6)

- [GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2505.20355/) (2025)
- [Understanding Post-Training Structural Changes in Large Language Models](https://mlsanity.com/paper/2509.17866/) (2025)
- [A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation](https://mlsanity.com/paper/2601.22708/) (2026)
- [PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspaces](https://mlsanity.com/paper/2609.04715/) (2026)
- [Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers](https://mlsanity.com/paper/2609.18056/) (2026)
- [What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation](https://mlsanity.com/paper/2609.32002/) (2026)

## Abstract (arXiv)

Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at: https://github.com/zwhe99/RaSA.