mlsanityData to 13 Oct 2026

AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning

arXiv 2403.09113 ↗First posted 14 Mar 2024cs.CL cs.AI cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA's uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA.

Key facts. Cited by 30 papers in the map (data through 13 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

25 of 43 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,870
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  11. 2018On First-Order Meta-Learning Algorithmscited by 1,010
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 891
13 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 778
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 665
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 459
  4. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 378
  5. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 359
  6. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  7. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  8. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
  9. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  10. 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
  11. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
  12. 2023PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspacescited by 15
  13. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 11.

30 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 652
  2. 2024ReFT: Representation Finetuning for Language Modelscited by 191
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 35
  5. 2025Improved Representation Steering for Language Modelscited by 19
  6. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
  7. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  8. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  9. 2026Why Steering Works: Toward a Unified View of Language Model Parameter Dynamicscited by 9
  10. 2024Transfer Learning for Finetuning Large Language Modelscited by 5
  11. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  12. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
18 more
  1. 2024Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Compositioncited by 3
  2. 2026ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimationcited by 3
  3. 2024Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiencycited by 2
  4. 2025SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformerscited by 2
  5. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  6. 2025Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuningcited by 2
  7. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  8. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  9. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 2
  10. 2024TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretrainingcited by 1
  11. 2025DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selectioncited by 1
  12. 2025LoRMA: Low-Rank Multiplicative Adaptation for LLMscited by 1
  13. 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
  14. 2026FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selectioncited by 0
  15. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0
  16. 2026MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuningcited by 0
  17. 2026Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Trainingcited by 0
  18. 2026DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformerscited by 0