# Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer

arXiv: https://arxiv.org/abs/2502.12964 · HTML: https://mlsanity.com/paper/2502.12964/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Truth is Universal: Robust Detection of Lies in LLMs](https://mlsanity.com/paper/2407.12831/) (2024)
- [Do Androids Know They're Only Dreaming of Electric Sheep?](https://mlsanity.com/paper/2312.17249/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations](https://mlsanity.com/paper/2404.10960/) (2024)
- [Personas as a Way to Model Truthfulness in Language Models](https://mlsanity.com/paper/2310.18168/) (2023)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty](https://mlsanity.com/paper/2408.06816/) (2024)
- [WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild](https://mlsanity.com/paper/2409.03753/) (2024)

## Built on by (most cited first, 27 of 27)

- [Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.09853/) (2025)
- [United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory](https://mlsanity.com/paper/2506.06843/) (2025)
- [BaseCal: Unsupervised Confidence Calibration via Base Model Signals](https://mlsanity.com/paper/2601.03042/) (2026)
- [DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching](https://mlsanity.com/paper/2511.00640/) (2025)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [SEReDeEP: Hallucination Detection in Retrieval-Augmented Models via Semantic Entropy and Context-Parameter Fusion](https://mlsanity.com/paper/2505.07528/) (2025)
- [Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling](https://mlsanity.com/paper/2607.01612/) (2026)
- [TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG](https://mlsanity.com/paper/2512.07515/) (2025)
- [Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer](https://mlsanity.com/paper/2605.22007/) (2026)
- [Quantifying Faithful Confidence Expression in Large Reasoning Models](https://mlsanity.com/paper/2606.03969/) (2026)
- [Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models](https://mlsanity.com/paper/2511.11751/) (2025)
- [Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"](https://mlsanity.com/paper/2602.04853/) (2026)
- [Sigma-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2607.27958/) (2026)
- [Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance](https://mlsanity.com/paper/2608.06539/) (2026)
- [Temporal Multi-Signal Fusion for Token-Level Hallucination Detection](https://mlsanity.com/paper/2608.18115/) (2026)
- [SABER: Stability-Aware Early Exit for LLM Reasoning via Adversarial Branch Probing](https://mlsanity.com/paper/2608.27963/) (2026)
- [Tractable Asymmetric Verification for Large Language Models via Deterministic Replicability](https://mlsanity.com/paper/2509.11068/) (2025)
- [Aletheia: Quantifying Cognitive Conviction in Reasoning Models via Regularized Inverse Confusion Matrix](https://mlsanity.com/paper/2601.01532/) (2026)
- [Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs](https://mlsanity.com/paper/2604.02230/) (2026)
- [Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments](https://mlsanity.com/paper/2605.03971/) (2026)
- [Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering](https://mlsanity.com/paper/2605.19220/) (2026)
- [The Role of Ambiguity in Error Prediction via Uncertainty Quantification](https://mlsanity.com/paper/2606.02093/) (2026)
- [Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Models](https://mlsanity.com/paper/2607.18292/) (2026)
- [Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows](https://mlsanity.com/paper/2608.05602/) (2026)
- [TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs](https://mlsanity.com/paper/2608.05616/) (2026)
- [The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining](https://mlsanity.com/paper/2609.32964/) (2026)
- [Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation](https://mlsanity.com/paper/2609.35308/) (2026)

## Abstract (arXiv)

Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety. The code is available at https://github.com/technion-cs-nlp/Trust_me_Im_wrong .