mlsanityData to 7 Oct 2026

Improving LoRA with Variational Learning

arXiv 2506.14280 ↗First posted 17 Jun 2025cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can sometimes even be detrimental. Moreover, Bayesian methods also increase computational overheads and require additional tricks for them to work well. Here, we fix these issues by using a recently proposed variational algorithm called IVON. We show that IVON is easy to implement and has similar costs to AdamW, and yet it can also drastically improve many metrics by using a simple posterior pruning technique. We present extensive results on billion-scale LLMs (Llama and Qwen series) going way beyond the scale of existing applications of IVON. For example, we finetune a Llama-3.2-3B model on a set of commonsense reasoning tasks and improve accuracy over AdamW by 1.3% and reduce ECE by 5.4%, outperforming AdamW and other recent Bayesian methods like Laplace-LoRA and BLoB. Overall, our results show that variational learning with IVON can effectively improve LoRA finetuning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

43 of 56 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
31 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  4. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  5. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  6. 2024LoRA Learns Less and Forgets Lesscited by 313
  7. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  8. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  9. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  10. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  11. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  12. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  13. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  14. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  15. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  16. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  17. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  18. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  19. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  20. 2023LoRA ensembles for large language model fine-tuningcited by 65
  21. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  22. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  23. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  24. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  25. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 19
  26. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  27. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  28. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  29. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  30. 2021Low-rank variational Bayes correction to the Laplace methodcited by 2
  31. 2024A Bayesian Interpretation of Adaptive Low-Rank Adaptationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Variational Low-Rank Adaptation for Personalized Impaired Speech Recognitioncited by 3
  2. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 1
  3. 2026Uncertainty-Aware Generation and Decision-Making Under Ambiguitycited by 1
  4. 2026Parameter Exploration for RLVR via Variational Learningcited by 0
  5. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
  2. 2025LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
  3. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
  4. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
  5. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
  6. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  8. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently