# Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes

arXiv: https://arxiv.org/abs/2507.00327 · HTML: https://mlsanity.com/paper/2507.00327/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [ReLoRA: High-Rank Training Through Low-Rank Updates](https://mlsanity.com/paper/2307.05695/) (2023)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning](https://mlsanity.com/paper/2401.04151/) (2024)
- [Stable Rank Normalization for Improved Generalization in Neural Networks and GANs](https://mlsanity.com/paper/1906.04659/) (2019)
- [MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification](https://mlsanity.com/paper/2306.09579/) (2023)
- [QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation](https://mlsanity.com/paper/2406.00132/) (2024)
- [GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning](https://mlsanity.com/paper/2412.09250/) (2024)
- [Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2407.01320/) (2024)

## Built on by (most cited first, 3 of 3)

- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation](https://mlsanity.com/paper/2607.09757/) (2026)
- [AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning](https://mlsanity.com/paper/2608.23816/) (2026)

## Abstract (arXiv)

Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models across various tasks. However, its fixed low-rank structure restricts its adaptability in scenarios with substantial domain gaps, where higher ranks are often required to capture domain-specific complexities. Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks, but these approaches frequently depend on computationally intensive techniques such as iterative pruning, rank searches, or additional regularization. To address these challenges, we introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation. By leveraging the stable rank, which reflects the intrinsic dimensionality of the weights, SR-LoRA enables a principled and efficient redistribution of ranks across layers, enhancing adaptability without incurring additional search costs. Empirical evaluations on few-shot tasks with significant domain gaps show that SR-LoRA consistently outperforms recent adaptive LoRA variants, achieving a superior trade-off between performance and efficiency. Our code is available at https://github.com/EndoluminalSurgicalVision-IMR/SR-LoRA.