mlsanityData to 7 Oct 2026

Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

arXiv 2405.03425 ↗First posted 6 May 2024cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets. To address these challenges, we propose a simple combination of Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG), facilitating approximate Bayesian inference in LLMs. Through extensive testing across several Natural Language Processing (NLP) benchmarks, we demonstrate that our straightforward and computationally efficient approach improves model generalization and calibration competitively with comparable, more sophisticated methods for Bayesian inference in LLMs. We further show that our method exhibits greater robustness against distribution shift, as reflected in its improved performance on out-of-distribution tasks.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 69 of its 86 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

69 of 86 references.

References · 69

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  9. 2015Weight Uncertainty in Neural Networkscited by 554
  10. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  11. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  13. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  14. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  15. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  16. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  17. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  18. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  19. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  20. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  21. 2019Ensemble Distribution Distillationcited by 158
  22. 2019Practical Deep Learning with Bayesian Principlescited by 158
  23. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  24. 2021Bayesian Neural Network Priors Revisitedcited by 121
  25. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  26. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  27. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  28. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  29. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  30. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  31. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  32. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  33. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  34. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  35. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  36. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  37. 2023LoRA ensembles for large language model fine-tuningcited by 65
  38. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  2. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  3. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  4. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  5. 2025Improving LoRA with Variational Learningcited by 5
  6. 2024Uncertainty-Guided Likelihood Tree Searchcited by 4
  7. 2025Bayesian Mixture of Experts For Large Language Modelscited by 4
  8. 2024Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimationcited by 3
  9. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  10. 2025Uncertainty-Aware Adaptation of Large Language Models for Protein-Protein Interaction Analysiscited by 2
  11. 20251LoRA: Summation Compression for Very Low-Rank Adaptationcited by 2
  12. 2025Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillationcited by 2
8 more
  1. 2026Calibrating Scientific Foundation Models with Inference-Time Stochastic Attentioncited by 1
  2. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1
  3. 2024Stein Variational Newton Neural Network Ensemblescited by 0
  4. 2025Learning Adapter Rank via Symmetry Breakingcited by 0
  5. 2026Bayesian Fine-tuning in Projected Subspacescited by 0
  6. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0
  7. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0
  8. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0