mlsanityData to 7 Oct 2026

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

arXiv 2303.10512 ↗First posted 18 Mar 2023cs.CL cs.LGcited by 337

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at https://github.com/QingruZhang/AdaLoRA .

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 18 Mar 2023.

Builds on

29 of 36 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2016Layer Normalizationcited by 5,579
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
17 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2017Learning multiple visual domains with residual adapterscited by 553
  11. 2019Importance Estimation for Neural Network Pruningcited by 531
  12. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  13. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  14. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  15. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  16. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  17. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 64; 2025: 129; 2026: 142.

337 papers.

Sort:

By area:

  1. 2024Chronos: Learning the Language of Time Seriescited by 712
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  4. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  5. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  6. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  8. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  9. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  10. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  11. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  12. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
38 more
  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  2. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  3. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  4. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  5. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  6. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  7. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  8. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 21
  9. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  10. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  11. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  12. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  13. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  14. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  15. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  16. 2025Improved Representation Steering for Language Modelscited by 17
  17. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 16
  18. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  19. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  20. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  21. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  22. 2024Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuningcited by 14
  23. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  24. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  25. 2026Learning to Reason in 13 Parameterscited by 13
  26. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  27. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  28. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
  29. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  30. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  31. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  32. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  33. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Productcited by 10
  34. 2024Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Taskscited by 9
  35. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Modelscited by 9
  36. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  37. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
  38. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9

50 most cited of 337 All 337 (JSON)