mlsanityData to 14 Oct 2026

Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations

arXiv 2404.10960 ↗First posted 16 Apr 2024cs.CL cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don't know. Inspired by analogous approaches in classification, this study explores the feasibility and efficacy of abstaining while uncertain in the context of LLMs within the domain of question-answering. We investigate two kinds of uncertainties, statistical uncertainty metrics and a distinct verbalized measure, termed as In-Dialogue Uncertainty (InDU). Using these uncertainty measures combined with models with and without Reinforcement Learning with Human Feedback (RLHF), we show that in all three situations, abstention based on the right kind of uncertainty measure can boost the reliability of LLMs. By sacrificing only a few highly uncertain samples we can improve correctness by 2% to 8%, avoid 50% hallucinations via correctly identifying unanswerable questions and increase safety by 70% up to 99% with almost no additional computational overhead.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

33 of 46 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  10. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
21 more
  1. 2017Selective Classification for Deep Neural Networkscited by 852
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  7. 2018CoQA: A Conversational Question Answering Challengecited by 683
  8. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  10. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  11. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  12. 2023Do Large Language Models Know What They Don't Know?cited by 243
  13. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  14. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  15. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 61
  16. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 55
  17. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  18. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
  19. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 27
  20. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  21. 2023"You might think about slightly revising the title": identifying hedges in peer-tutoring interactionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 21; 2026: 25.

54 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  4. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  5. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  6. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
  7. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  8. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  9. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  10. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 27
  11. 2024A Survey on the Honesty of Large Language Modelscited by 23
  12. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
38 more
  1. 2026AgentAbstain: Do LLM Agents Know When Not to Act?cited by 17
  2. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertaintycited by 15
  3. 2025Query-Level Uncertainty in Large Language Modelscited by 14
  4. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  5. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 13
  6. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 12
  7. 2026Over-Searching in Search-Augmented Large Language Modelscited by 11
  8. 2026Causal Evidence that Language Models use Confidence to Drive Behaviorcited by 10
  9. 2026The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimizationcited by 8
  10. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  11. 2026When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?cited by 7
  12. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
  13. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 5
  14. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  15. 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 5
  16. 2024Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selectioncited by 4
  17. 2026Steering the Verifiability of Multimodal AI Hallucinationscited by 4
  18. 2026EquiMem: Calibrating Shared Memory in Multi-Agent Debate via Game-Theoretic Equilibriumcited by 4
  19. 2025Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Modelscited by 3
  20. 2025Knowing What's Missing: Assessing Information Sufficiency in Question Answeringcited by 3
  21. 2024Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questionscited by 2
  22. 2025R2-KG: General-Purpose Dual-Agent Framework for Reliable Reasoning on Knowledge Graphscited by 2
  23. 2025Know Or Not: a library for evaluating out-of-knowledge base robustnesscited by 2
  24. 2025Predicting Language Models' Success at Zero-Shot Probabilistic Predictioncited by 2
  25. 2026When Confidence Takes the Wrong Path: Diagnosing Retrieval-State Lock-In in RAGcited by 2
  26. 2026Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understandingcited by 2
  27. 2024Cognitively Inspired Energy-Based World Modelscited by 1
  28. 2026Probing for Knowledge Attribution in Large Language Modelscited by 1
  29. 2026No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generationcited by 1
  30. 2026Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queriescited by 1
  31. 2025Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrscited by 0
  32. 2025Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds in Language Modelscited by 0
  33. 2026RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMscited by 0
  34. 2026From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completioncited by 0
  35. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0
  36. 2026No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agentscited by 0
  37. 2026Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Modelscited by 0
  38. 2026Evaluating medical AI under missing information: same-provider judges and human raters change apparent safetycited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
  2. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
  3. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  4. 2025Teaching LLMs to Abstain via Fine-Grained Semantic Confidence Reward
  5. 2026Hallucinations Undermine Trust; Metacognition is a Way Forward
  6. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learning
  7. 2024Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation
  8. 2025Delusions of Large Language Models