mlsanityData to 7 Oct 2026

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

arXiv 2406.11675 ↗First posted 17 Jun 2024cs.LG cs.AI cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often suffer from overconfidence during inference, particularly when adapted to downstream domain-specific tasks with limited data. Previous work addresses this issue by employing approximate Bayesian estimation after the LLMs are trained, enabling them to quantify uncertainty. However, such post-training approaches' performance is severely limited by the parameters learned during training. In this paper, we go beyond post-training Bayesianization and propose Bayesian Low-Rank Adaptation by Backpropagation (BLoB), an algorithm that continuously and jointly adjusts both the mean and covariance of LLM parameters throughout the whole fine-tuning process. Our empirical results verify the effectiveness of BLoB in terms of generalization and uncertainty estimation, when evaluated on both in-distribution and out-of-distribution data.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 87 of its 123 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

87 of 123 references.

References · 87

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2016Concrete Problems in AI Safetycited by 1,582
  13. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  14. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  15. 2023PaLM 2 Technical Reportcited by 1,154
  16. 2021Ethical and social risks of harm from Language Modelscited by 957
  17. 2018Neural Network Acceptability Judgmentscited by 952
  18. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  19. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  20. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  21. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  22. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  23. 2021GPT Understands, Toocited by 715
  24. 2020Aligning AI With Shared Human Valuescited by 706
  25. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  26. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  27. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  28. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  29. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  30. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  31. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  32. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  33. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  34. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  35. 2023Do Large Language Models Know What They Don't Know?cited by 238
  36. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  37. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  38. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 20; 2026: 25.

48 papers.

Sort:

By area:

  1. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  2. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  3. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  4. 2025Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inferencecited by 11
  5. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
  6. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  7. 2025Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'cited by 8
  8. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  9. 2025Random-Set Large Language Modelscited by 7
  10. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
  11. 2025Improving LoRA with Variational Learningcited by 5
  12. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
36 more
  1. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  2. 2025Uncertainty-Aware Adaptation of Large Language Models for Protein-Protein Interaction Analysiscited by 2
  3. 2025PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generationcited by 2
  4. 2025Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillationcited by 2
  5. 2025Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimizationcited by 2
  6. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Modelscited by 2
  7. 2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
  8. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  9. 2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
  10. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  11. 2025Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learningcited by 1
  12. 2025Bayesian Low-Rank Factorization for Robust Model Adaptationcited by 1
  13. 2026Quantifying the Uncertainty of Foundation Models with Singular Value Ensemblescited by 1
  14. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillationcited by 1
  15. 2026Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisationcited by 1
  16. 2026CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distractioncited by 1
  17. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 1
  18. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1
  19. 2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimationcited by 1
  20. 2025Learning Adapter Rank via Symmetry Breakingcited by 0
  21. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  22. 2025Kalman Bayesian Transformercited by 0
  23. 2025As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Filescited by 0
  24. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Modelscited by 0
  25. 2026Efficient Credal Prediction through Decalibrationcited by 0
  26. 2026Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimationcited by 0
  27. 2026Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learningcited by 0
  28. 2026Bayesian Fine-tuning in Projected Subspacescited by 0
  29. 2026On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inferencecited by 0
  30. 2026iLoRA: Bayesian Low-Rank Adaptation with Latent Interaction Graphs for Microbiome Diagnosiscited by 0
  31. 2026Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Modelscited by 0
  32. 2026Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Modelscited by 0
  33. 2026Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descentcited by 0
  34. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  35. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0
  36. 2026Learning to Predict Distributions over Weight Updates for Test-Time Adaptationcited by 0