mlsanityData to 14 Oct 2026

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

arXiv 2601.21003 ↗First posted 28 Jan 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ≈1.2× training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

44 of 64 references.

References · 44

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2015Variational Inference with Normalizing Flowscited by 2,170
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  12. 2017Masked Autoregressive Flow for Density Estimationcited by 684
32 more
  1. 2015Weight Uncertainty in Neural Networkscited by 554
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  3. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  4. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  5. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  6. 2020Calibration of Pre-trained Transformerscited by 315
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  8. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  9. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  10. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  11. 2024To Believe or Not to Believe Your LLMcited by 120
  12. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  13. 2024Data-efficient Fine-tuning for LLM-based Recommendationcited by 104
  14. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  15. 2025Medical Hallucinations in Foundation Models and Their Impact on Healthcarecited by 93
  16. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  17. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 57
  18. 2020Convergence of Sparse Variational Inference in Gaussian Processes Regressioncited by 53
  19. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  20. 2025Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalizationcited by 29
  21. 2024Fine-Tuning is Fine, if Calibratedcited by 24
  22. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 23
  23. 2021Bayesian Transformer Language Models for Speech Recognitioncited by 22
  24. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  25. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  26. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 12
  27. 2025Uncertainty Quantification and Decomposition for LLM-based Recommendationcited by 12
  28. 2024DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Drivingcited by 11
  29. 2023DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibrationcited by 6
  30. 2025Ensemble based approach to quantifying uncertainty of LLM based classificationscited by 2
  31. 2025Stochastic Weight Sharing for Bayesian Neural Networkscited by 2
  32. 2025Flow-Induced Diagonal Gaussian Processescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  2. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
  2. 2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates
  3. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  4. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  6. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  7. 2026Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
  8. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models