mlsanityData to 7 Oct 2026

Uncertainty-aware Language Modeling for Selective Question Answering

arXiv 2311.15451 ↗First posted 26 Nov 2023cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We present an automatic large language model (LLM) conversion approach that produces uncertainty-aware LLMs capable of estimating uncertainty with every prediction. Our approach is model- and data-agnostic, is computationally-efficient, and does not rely on external models or systems. We evaluate converted models on the selective question answering setting -- to answer as many questions as possible while maintaining a given accuracy, forgoing providing predictions when necessary. As part of our results, we test BERT and Llama 2 model variants on the SQuAD extractive QA task and the TruthfulQA generative QA task. We show that using the uncertainty estimates provided by our approach to selectively answer questions leads to significantly higher accuracy over directly using model probabilities.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

23 of 35 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2017Selective Classification for Deep Neural Networkscited by 799
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2021A Survey on Automated Fact-Checkingcited by 376
11 more
  1. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  2. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  3. 2020Selective Question Answering under Domain Shiftcited by 228
  4. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  5. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  6. 2019DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networkscited by 92
  7. 2019Quizbowl: The Case for Incremental Question Answeringcited by 36
  8. 2018Confidence Modeling for Neural Semantic Parsingcited by 33
  9. 2022Post-hoc Uncertainty Learning using a Dirichlet Meta-Modelcited by 31
  10. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  11. 2022Improving the repeatability of deep learning models with Monte Carlo dropoutcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Towards Conversational Diagnostic AIcited by 79
  2. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  3. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  4. 2024Multicalibration for Confidence Scoring in LLMscited by 44
  5. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 23
  6. 2024Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answerscited by 18
  7. 2024CLUE: Concept-Level Uncertainty Estimation for Large Language Modelscited by 4
  8. 2025Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMscited by 3
  9. 2025Model-agnostic Selective Labeling with Provable Statistical Guaranteescited by 2
  10. 2026Uncertainty and Fairness Awareness in LLM-Based Recommendation Systemscited by 1
  11. 2025Zero-Shot Event Causality Identification via Multi-source Evidence Fuzzy Aggregation with Large Language Modelscited by 0
  12. 2025A Bayesian Hybrid Parameter-Efficient Fine-Tuning Method for Large Language Modelscited by 0
4 more
  1. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  2. 2026Evaluating the Relevance of Uncertainty Estimators for LLM Hallucinationcited by 0
  3. 2026Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splittingcited by 0
  4. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0