mlsanityData to 7 Oct 2026

Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations

arXiv 2503.14477 ↗First posted 18 Mar 2025cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

LLMs often adopt an assertive language style also when making false claims. Such ``overconfident hallucinations'' mislead users and erode trust. Achieving the ability to express in language the actual degree of uncertainty around a claim is therefore of great importance. We find that ``verbal uncertainty'' is governed by a single linear feature in the representation space of LLMs, and show that this has only moderate correlation with the actual ``semantic uncertainty'' of the model. We apply this insight and show that (1) the mismatch between semantic and verbal uncertainty is a better predictor of hallucinations than semantic uncertainty alone and (2) we can intervene on verbal uncertainty at inference time and reduce confident hallucinations on short-form answers, achieving an average relative reduction of ~30%.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 64 of its 80 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

64 of 80 references.

References · 64

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2023Mistral 7Bcited by 3,065
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
38 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2022Toy Models of Superpositioncited by 928
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  9. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  13. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  14. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  15. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  16. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  17. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  18. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  19. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  20. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  21. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  22. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  23. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  24. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  25. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  26. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  27. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  28. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  29. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  30. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  31. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  32. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  33. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  34. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  35. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  36. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  37. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  38. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 30.

48 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  2. 2025Embodied AI Agents: Modeling the Worldcited by 57
  3. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  4. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 22
  5. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  6. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  7. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  8. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  9. 2026AgentAbstain: Do LLM Agents Know When Not to Act?cited by 8
  10. 2025Teaching Language Models to Faithfully Express their Uncertaintycited by 7
  11. 2025Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learningcited by 6
  12. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
36 more
  1. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  2. 2026Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibrationcited by 6
  3. 2025Can Large Language Models Express Uncertainty Like Human?cited by 5
  4. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  5. 2026Steering the Verifiability of Multimodal AI Hallucinationscited by 4
  6. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  7. 2026The Value Axis: Language Models Encode Whether They're on the Right Trackcited by 4
  8. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  9. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 3
  10. 2025GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language Modelscited by 3
  11. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 3
  12. 2025Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processescited by 3
  13. 2025SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstentioncited by 2
  14. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
  15. 2026Retrieval-Augmented Linguistic Calibrationcited by 2
  16. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  17. 2026Quantifying Faithful Confidence Expression in Large Reasoning Modelscited by 2
  18. 2026A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluationcited by 2
  19. 2026When Linguistic and Internal Confidence Diverge in Large Language Modelscited by 2
  20. 2025Explaining Sources of Uncertainty in Automated Fact-Checkingcited by 1
  21. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 1
  22. 2026Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learningcited by 1
  23. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  24. 2026Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoningcited by 1
  25. 2026Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Surveycited by 0
  26. 2026Identifying Influential N-grams in Confidence Calibration via Regression Analysiscited by 0
  27. 2026Architecture Determines Observability of Transformerscited by 0
  28. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  29. 2026RepBench: Compiling Benchmarks into Capability Representations for Large Language Modelscited by 0
  30. 2026Vision-Language Model Confidence Is Not a Property of the Answercited by 0
  31. 2026Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoningcited by 0
  32. 2026Different Facets of Verbalised Overconfidence: an Interpretability Studycited by 0
  33. 2026The Halt Vector: Internalizing a Causal Steering Intervention for Efficient Reasoningcited by 0
  34. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Modelscited by 0
  35. 2026"very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantificationcited by 0
  36. 2026Verbalized and Internal Probabilities Are Coupled in Large Language Modelscited by 0