mlsanityData to 7 Oct 2026

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

arXiv 2403.16187 ↗First posted 24 Mar 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. First, we propose a novel method, AB-LoRA, that can effectively estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we gradually prune abundant and negatively impacting LoRA ranks and allocate the pruned LoRA budgets to important Transformer modules needing higher ranks. We have conducted experiments on various tasks, and the experimental results demonstrate that our ALoRA method can outperform the recent baselines with comparable tunable parameters.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 50 of its 89 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

50 of 89 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  13. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  14. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  15. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  16. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  17. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  18. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  19. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  20. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  21. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  22. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  23. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  24. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  25. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  26. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  27. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  28. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  29. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  30. 2019Progressive DARTS: Bridging the Optimization Gap for NAS in the Wildcited by 39
  31. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  32. 2022Late Prompt Tuning: A Late Prompt Could Be Better Than Many Promptscited by 23
  33. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  34. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 16
  35. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  36. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  37. 2020AutoTrans: Automating Transformer Design via Reinforced Architecture Searchcited by 12
  38. 2020AutoRC: Improving BERT Based Relation Classification Models via Architecture Searchcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 18; 2026: 10.

36 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 21
  3. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  4. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  5. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  6. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  8. 2024Sparse Matrix in Large Language Model Fine-tuningcited by 8
  9. 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenescited by 8
  10. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  11. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  12. 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
24 more
  1. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  2. 2026IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoringcited by 5
  3. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  4. 2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
  5. 2025WeightLoRA: Keep Only Necessary Adapterscited by 3
  6. 2026ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimationcited by 3
  7. 2024SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesiscited by 2
  8. 2024VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognitioncited by 2
  9. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  10. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  11. 2025DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selectioncited by 1
  12. 2025Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learningcited by 1
  13. 2025QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Modelscited by 1
  14. 2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimationcited by 1
  15. 2024ChildDiffusion: Unlocking the Potential of Generative AI and Controllable Augmentations for Child Facial Data using Stable Diffusion and Large Language Modelscited by 0
  16. 2025Exploring Adapter Design Tradeoffs for Low Resource Music Generationcited by 0
  17. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  18. 2025GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptationcited by 0
  19. 2025Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptationcited by 0
  20. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0
  21. 2026Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigationcited by 0
  22. 2026FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimationcited by 0
  23. 2026Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Trainingcited by 0
  24. 2026IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuningcited by 0