mlsanityData to 14 Oct 2026

ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs

arXiv 2506.18267 ↗First posted 23 Jun 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Conventional Low-Rank Adaptation (LoRA) methods employ a fixed rank, imposing uniform adaptation across transformer layers and attention heads despite their heterogeneous learning dynamics. This paper introduces Adaptive Rank Dynamic LoRA (ARD-LoRA), a novel framework that automates rank allocation through learnable scaling factors. These factors are optimized via a meta-objective balancing task performance and parameter efficiency, incorporating ℓ1 sparsity for minimal rank and Total Variation regularization for stable rank transitions. ARD-LoRA enables continuous, differentiable, per-head rank adaptation. Experiments on LLAMA-3.1-70B and PaliGemma-2 demonstrate ARD-LoRA's efficacy, achieving up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters, outperforming strong baselines like DoRA and AdaLoRA. Furthermore, it reduces multimodal adaptation memory by 41%. These results establish dynamic, fine-grained rank allocation as a critical paradigm for efficient foundation model adaptation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

21 of 44 references.

References · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
9 more
  1. 2016Visual Dialogcited by 377
  2. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  3. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  4. 2024Higher Layers Need More LoRA Expertscited by 102
  5. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  6. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  7. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  8. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  9. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  2. 2026Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generationcited by 2
  3. 2026AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discoverycited by 0
  4. 2026LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 0
  5. 2026Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Modelscited by 0