# Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging

arXiv: https://arxiv.org/abs/2304.04726 · HTML: https://mlsanity.com/paper/2304.04726/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [When Do Flat Minima Optimizers Work?](https://mlsanity.com/paper/2202.00661/) (2022)
- [Uncertain Natural Language Inference](https://mlsanity.com/paper/1909.03042/) (2019)
- [Testing the Generalization Power of Neural Network Models Across NLI Benchmarks](https://mlsanity.com/paper/1810.09774/) (2018)
- [Revisiting Checkpoint Averaging for Neural Machine Translation](https://mlsanity.com/paper/2210.11803/) (2022)
- [Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging](https://mlsanity.com/paper/2212.05956/) (2022)
- [How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task](https://mlsanity.com/paper/2111.09612/) (2021)

## Built on by (most cited first, 4 of 4)

- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [Robust Hallucination Detection in LLMs via Adaptive Token Selection](https://mlsanity.com/paper/2504.07863/) (2025)
- [UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformers](https://mlsanity.com/paper/2602.02952/) (2026)
- [Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization](https://mlsanity.com/paper/2412.16232/) (2024)

## Abstract (arXiv)

This paper introduces Bayesian uncertainty modeling using Stochastic Weight Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We apply the approach to standard tasks in natural language inference (NLI) and demonstrate the effectiveness of the method in terms of prediction accuracy and correlation with human annotation disagreements. We argue that the uncertainty representations in SWAG better reflect subjective interpretation and the natural variation that is also present in human language understanding. The results reveal the importance of uncertainty modeling, an often neglected aspect of neural language modeling, in NLU tasks.