mlsanityData to 7 Oct 2026

Variational Low-Rank Adaptation Using IVON

arXiv 2411.04421 ↗First posted 7 Nov 2024cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

17 of 24 references.

References · 17

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  8. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  9. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  10. 2023LoRA ensembles for large language model fine-tuningcited by 65
  11. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  12. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
5 more
  1. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  2. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  3. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  4. 2021Low-rank variational Bayes correction to the Laplace methodcited by 2
  5. 2024A Bayesian Interpretation of Adaptive Low-Rank Adaptationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  2. 2025CLEAR: Calibrated Learning for Epistemic and Aleatoric Riskcited by 6
  3. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  4. 2026Calibrating Scientific Foundation Models with Inference-Time Stochastic Attentioncited by 1
  5. 2025Enhancing Collective Intelligence in Large Language Models Through Emotional Integrationcited by 0