mlsanityData to 7 Oct 2026

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models

arXiv 2412.05723 ↗First posted 7 Dec 2024stat.ML cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Estimating the uncertainty of responses from Large Language Models (LLMs) remains a critical challenge. While recent Bayesian methods have demonstrated effectiveness in quantifying uncertainty through low-rank weight updates, they typically require complex fine-tuning or post-training procedures. In this paper, we propose Training-Free Bayesianization (TFB), a simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within a family of low-rank isotropic Gaussian distributions. Our theoretical analysis shows that under mild conditions, this search process is equivalent to KL-regularized variational optimization, a generalized form of variational inference. Through comprehensive experiments, we show that TFB achieves superior uncertainty estimation and generalization compared to existing methods while eliminating the need for complex Bayesianization training procedures. Code will be available at https://github.com/Wang-ML-Lab/bayesian-peft.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

50 of 69 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  11. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  13. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  14. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  15. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  16. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  17. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  18. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  19. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  20. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  21. 2024To Believe or Not to Believe Your LLMcited by 120
  22. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  23. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  24. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  25. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  26. 2024Variational Bayesian Last Layerscited by 67
  27. 2023LoRA ensembles for large language model fine-tuningcited by 65
  28. 2016Towards Bayesian Deep Learning: A Framework and Some Existing Methodscited by 57
  29. 2019Generalized Variational Inference: Three arguments for deriving new Posteriorscited by 57
  30. 2020Bayesian Attention Modulescited by 51
  31. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  32. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  33. 2016Natural-Parameter Networks: A Class of Probabilistic Neural Networkscited by 33
  34. 2021Bayesian Attention Belief Networkscited by 25
  35. 2018Piecewise Strong Convexity of Neural Networkscited by 13
  36. 2024Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Modelscited by 6
  37. 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
  38. 2024Variational Language Concepts for Interpreting Foundation Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  2. 2025Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillationcited by 2
  3. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Modelscited by 2
  4. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  5. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  6. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillationcited by 1
  7. 2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimationcited by 1
  8. 2026Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Modelscited by 0
  9. 2026Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Modelscited by 0
  10. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0