# ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

arXiv: https://arxiv.org/abs/2403.16187 · HTML: https://mlsanity.com/paper/2403.16187/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 89 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)

## Built on by (most cited first, 30 of 36)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases](https://mlsanity.com/paper/2509.25238/) (2025)
- [FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.02961/) (2025)
- [S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity](https://mlsanity.com/paper/2412.06289/) (2024)
- [Dynamic Low-Rank Sparse Adaptation for Large Language Models](https://mlsanity.com/paper/2502.14816/) (2025)
- [DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation](https://mlsanity.com/paper/2503.18225/) (2025)
- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenes](https://mlsanity.com/paper/2509.16415/) (2025)
- [Sparse Matrix in Large Language Model Fine-tuning](https://mlsanity.com/paper/2405.15525/) (2024)
- [GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning](https://mlsanity.com/paper/2412.09250/) (2024)
- [Layer-wise dynamic rank for compressing large language models](https://mlsanity.com/paper/2509.25622/) (2025)
- [All You Need is One: Capsule Prompt Tuning with a Single Vector](https://mlsanity.com/paper/2510.16670/) (2025)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring](https://mlsanity.com/paper/2603.13792/) (2026)
- [Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts](https://mlsanity.com/paper/2506.21035/) (2025)
- [WeightLoRA: Keep Only Necessary Adapters](https://mlsanity.com/paper/2506.02724/) (2025)
- [FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts](https://mlsanity.com/paper/2506.00495/) (2025)
- [ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimation](https://mlsanity.com/paper/2601.17921/) (2026)
- [SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis](https://mlsanity.com/paper/2408.07196/) (2024)
- [VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition](https://mlsanity.com/paper/2412.20064/) (2024)
- [DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection](https://mlsanity.com/paper/2502.08905/) (2025)
- [ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints](https://mlsanity.com/paper/2507.08044/) (2025)
- [A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation](https://mlsanity.com/paper/2601.22708/) (2026)
- [Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning](https://mlsanity.com/paper/2505.22358/) (2025)
- [QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2508.21810/) (2025)
- [FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation](https://mlsanity.com/paper/2605.16800/) (2026)
- [Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation](https://mlsanity.com/paper/2607.02182/) (2026)
- [ChildDiffusion: Unlocking the Potential of Generative AI and Controllable Augmentations for Child Facial Data using Stable Diffusion and Large Language Models](https://mlsanity.com/paper/2406.11592/) (2024)
- [Exploring Adapter Design Tradeoffs for Low Resource Music Generation](https://mlsanity.com/paper/2506.21298/) (2025)
- [Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization](https://mlsanity.com/paper/2507.01841/) (2025)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. First, we propose a novel method, AB-LoRA, that can effectively estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we gradually prune abundant and negatively impacting LoRA ranks and allocate the pruned LoRA budgets to important Transformer modules needing higher ranks. We have conducted experiments on various tasks, and the experimental results demonstrate that our ALoRA method can outperform the recent baselines with comparable tunable parameters.