mlsanityData to 7 Oct 2026

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning

arXiv 2402.10462 ↗First posted 16 Feb 2024cs.LG cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

18 of 21 references.

References · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
6 more
  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  2. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  3. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  4. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  5. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  6. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 7; 2026: 10.

26 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  5. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  6. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  7. 2024Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalizationcited by 5
  8. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  9. 2025ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needscited by 5
  10. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  11. 2026Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learningcited by 4
  12. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 3
14 more
  1. 2024EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Modelscited by 2
  2. 2025MoKA: Mixture of Kronecker Adapterscited by 2
  3. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  4. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  5. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanismcited by 1
  6. 2025FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregationcited by 1
  7. 2025DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptationcited by 1
  8. 2025Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Modelscited by 1
  9. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inferencecited by 1
  10. 2026Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgettingcited by 1
  11. 2025Language-Aware Information Maximization for Transductive Few-Shot CLIPcited by 0
  12. 2026Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learningcited by 0
  13. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0
  14. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0