mlsanityData to 7 Oct 2026

A Bayesian Interpretation of Adaptive Low-Rank Adaptation

arXiv 2409.10673 ↗First posted 16 Sep 2024cs.LG cs.CL stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Motivated by the sensitivity-based importance score of the adaptive low-rank adaptation (AdaLoRA), we utilize more theoretically supported metrics, including the signal-to-noise ratio (SNR), along with the Improved Variational Online Newton (IVON) optimizer, for adaptive parameter budget allocation. The resulting Bayesian counterpart not only has matched or surpassed the performance of using the sensitivity-based importance metric but is also a faster alternative to AdaLoRA with Adam. Our theoretical analysis reveals a significant connection between the two metrics, providing a Bayesian perspective on the efficacy of sensitivity as an importance score. Furthermore, our findings suggest that the magnitude, rather than the variance, is the primary indicator of the importance of parameters.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

18 of 24 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2019Importance Estimation for Neural Network Pruningcited by 531
  8. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  9. 2019Practical Deep Learning with Bayesian Principlescited by 158
  10. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  11. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  12. 2024Variational Learning is Effective for Large Deep Networkscited by 62
6 more
  1. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  2. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  3. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  4. 2017Structured Bayesian Pruning via Log-Normal Multiplicative Noisecited by 39
  5. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 20
  6. 2024Training Bayesian Neural Networks with Sparse Subspace Variational Inferencecited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  2. 2025Improving LoRA with Variational Learningcited by 5