Data to 7 Oct 2026

Bayesian Low-rank Adaptation for Large Language Models

arXiv 2308.13111 ↗First posted 24 Aug 2023cs.LGcited by 112

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has emerged as a new paradigm for cost-efficient fine-tuning of large language models (LLMs). However, fine-tuned LLMs often become overconfident especially when fine-tuned on small datasets. Bayesian methods, with their inherent ability to estimate uncertainty, serve as potent tools to mitigate overconfidence and enhance calibration. In this work, we introduce Laplace-LoRA, which applies a Bayesian approach to the LoRA parameters. Specifically, Laplace-LoRA applies a Laplace approximation to the posterior over the LoRA parameters, considerably improving the calibration of fine-tuned LLMs.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

57 of 74 references.

References · 57

2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2020Measuring Massive Multitask Language Understandingcited by 7,768
2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
2017On Calibration of Modern Neural Networkscited by 4,860
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195

38 more

2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
2022Large Language Models Encode Clinical Knowledgecited by 1,807
2022Language Models (Mostly) Know What They Knowcited by 1,783
2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
2023BloombergGPT: A Large Language Model for Financecited by 923
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
2021Bayesian Neural Network Priors Revisitedcited by 121
2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
2019A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inferencecited by 51
2020Bayesian Attention Modulescited by 51
2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
2020Deep kernel processescited by 34
2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 27
2021Bayesian Attention Belief Networkscited by 25

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 31; 2025: 39; 2026: 40.

112 papers.

Sort:

By area:

2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
2024A Survey on LoRA of Large Language Modelscited by 67
2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
2024Instruction Tuning With Loss Over Instructionscited by 40
2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35

38 more

2024Bayesian Reward Models for LLM Alignmentcited by 35
2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
2024Post-hoc Probabilistic Vision-Language Modelscited by 13
2025Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inferencecited by 11
2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 10
2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
2025Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'cited by 8
2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
2025Random-Set Large Language Modelscited by 7
2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2024Variational Low-Rank Adaptation Using IVONcited by 5
2024Streamlining Prediction in Bayesian Deep Learningcited by 5
2025BARNN: A Bayesian Autoregressive and Recurrent Neural Networkcited by 5
2025Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRAcited by 5
2025Improving LoRA with Variational Learningcited by 5
2024Uncertainty-Guided Likelihood Tree Searchcited by 4
2025Bayesian Mixture of Experts For Large Language Modelscited by 4
2024On Uncertainty Quantification for Near-Bayes Optimal Algorithmscited by 3
2024Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Imagescited by 3
2024SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptationcited by 3
2024Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimationcited by 3
2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3

50 most cited of 112 All 112 (JSON)