mlsanityData to 14 Oct 2026

Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation

arXiv 2607.02182 ↗First posted 2 Jul 2026cs.LG cs.CLcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment. We propose Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty quantification from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA). With the insight that LoRA essentially aggregates multiple rank-one components that may provide superfluous model capacity, DALorRA imposes stochastic masking on rank dimensions, enabling Bayesian regularization of model capacity during training and ensemble-like calibration during inference. Extensive experiments demonstrate DALorRA's excellent calibration of LLMs without compromising reasoning accuracy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 38 of its 52 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

38 of 52 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2013Auto-Encoding Variational Bayescited by 4,053
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
26 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  2. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  3. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  4. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  5. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  6. 2023LoRA ensembles for large language model fine-tuningcited by 67
  7. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  8. 2014Exact Post-Selection Inference for Sequential Regression Procedurescited by 48
  9. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37
  10. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  11. 2025Calibrating LLMs with Information-Theoretic Evidential Deep Learningcited by 15
  12. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 12
  13. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  14. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  15. 2025UQLM: A Python Package for Uncertainty Quantification in Large Language Modelscited by 9
  16. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  17. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 7
  18. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 6
  19. 2025Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRAcited by 5
  20. 2026Post-Optimization Adaptive Rank Allocation for LoRAcited by 4
  21. 2025Balancing Two Classifiers via A Simplex ETF Structure for Model Calibrationcited by 3
  22. 2024Non-Asymptotic Uncertainty Quantification in High-Dimensional Learningcited by 2
  23. 2025Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillationcited by 2
  24. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 2
  25. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  26. 2025Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Maskingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
  2. 2026Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
  3. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  4. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  5. 2025Bayesian Mixture of Experts For Large Language Models
  6. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Training
  7. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models
  8. 2026Filtering Beats Fine Tuning: A Bayesian Kalman View of In Context Learning in LLMs