Data to 7 Oct 2026

LoRA ensembles for large language model fine-tuning

arXiv 2310.00035 ↗First posted 29 Sep 2023cs.LG cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

Finetuned LLMs often exhibit poor uncertainty quantification, manifesting as overconfidence, poor calibration, and unreliable prediction results on test data or out-of-distribution samples. One approach commonly used in vision for alleviating this issue is a deep ensemble, which constructs an ensemble by training the same model multiple times using different random initializations. However, there is a huge challenge to ensembling LLMs: the most effective LLMs are very, very large. Keeping a single LLM in memory is already challenging enough: keeping an ensemble of e.g. 5 LLMs in memory is impossible in many settings. To address these issues, we propose an ensemble approach using Low-Rank Adapters (LoRA), a parameter-efficient fine-tuning technique. Critically, these low-rank adapters represent a very small number of parameters, orders of magnitude less than the underlying pre-trained model. Thus, it is possible to construct large ensembles of LoRA adapters with almost the same computational overhead as using the original model. We find that LoRA ensembles, applied on its own or on top of pre-existing regularization techniques, gives consistent improvements in predictive accuracy and uncertainty quantification.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

43 of 63 references.

References · 43

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2020Measuring Massive Multitask Language Understandingcited by 7,768
2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
2017On Calibration of Modern Neural Networkscited by 4,860
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416

31 more

2022Large Language Models are Zero-Shot Reasonerscited by 4,323
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
2022Large Language Models Encode Clinical Knowledgecited by 1,807
2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2022Red Teaming Language Models with Language Modelscited by 983
2022Teaching Models to Express Their Uncertainty in Wordscited by 731
2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
2017Snapshot Ensembles: Train 1, get M for freecited by 349
2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
2023FinGPT: Open-Source Financial Large Language Modelscited by 244
2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
2020Few-Shot Learning via Learning the Representation, Provablycited by 213
2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
2022Uncertainty Estimation for Language Reward Modelscited by 35
2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
2022BayesFormer: Transformer with Uncertainty Estimationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 22; 2025: 21; 2026: 20.

65 papers.

Sort:

By area:

2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
2024A Survey on LoRA of Large Language Modelscited by 67
2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43
2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
2024Bayesian Reward Models for LLM Alignmentcited by 35
2024On the Challenges and Opportunities in Generative AIcited by 31
2025Parallel Scaling Law for Language Modelscited by 29
2024Hyperbolic Fine-Tuning for Large Language Modelscited by 24
2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20

38 more

2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networkscited by 20
2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
2024Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Modelscited by 15
2023Large Language Models are Miscalibrated In-Context Learnerscited by 12
2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
2024LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Modelscited by 9
2025Ensembles of Low-Rank Expert Adapterscited by 9
2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
2024STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Modelscited by 5
2024Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learningcited by 5
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2024Variational Low-Rank Adaptation Using IVONcited by 5
2025Improving LoRA with Variational Learningcited by 5
2025Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformerscited by 5
2024Optimizing Large Language Models with an Enhanced LoRA Fine-Tuning Algorithm for Efficiency and Robustness in NLP Taskscited by 4
2025Bayesian Mixture of Experts For Large Language Modelscited by 4
2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
2024CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Modelscited by 2
2024Why you don't overfit, and don't need Bayes if you only train for one epochcited by 2
2025Uncertainty-Aware Adaptation of Large Language Models for Protein-Protein Interaction Analysiscited by 2
2025Investigating Human-Aligned Large Language Model Uncertaintycited by 2
2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
2025Neural Diversity Regularizes Hallucinations in Language Modelscited by 2
2026Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapterscited by 2
2026Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Modelscited by 2
2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
2025Beyond Negation Detection: Comprehensive Assertion Detection Models for Clinical NLPcited by 1
2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
2026Quantifying the Uncertainty of Foundation Models with Singular Value Ensemblescited by 1
2026BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Modelscited by 1
2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
2026Conf-Gen: Conformal Uncertainty Quantification for Generative Modelscited by 1
2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1
2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimationcited by 1
2024Bayesian Low-Rank LeArning (Bella): A Practical Approach to Bayesian Neural Networkscited by 0

50 most cited of 65 All 65 (JSON)