# Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models

arXiv: https://arxiv.org/abs/2603.15557 · HTML: https://mlsanity.com/paper/2603.15557/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 38)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024)
- [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://mlsanity.com/paper/2305.10400/) (2023)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection](https://mlsanity.com/paper/2409.17504/) (2024)
- [Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?](https://mlsanity.com/paper/2405.16908/) (2024)
- [Selectively Answering Ambiguous Questions](https://mlsanity.com/paper/2305.14613/) (2023)
- [Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training](https://mlsanity.com/paper/2403.02325/) (2024)
- [Multi-Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2407.06192/) (2024)
- [Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations](https://mlsanity.com/paper/2503.14477/) (2025)
- [How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts](https://mlsanity.com/paper/2402.13220/) (2024)
- [Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models](https://mlsanity.com/paper/2307.10236/) (2023)
- [Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization](https://mlsanity.com/paper/2405.15356/) (2024)
- [VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues](https://mlsanity.com/paper/2502.12084/) (2025)
- [Estimating the Hallucination Rate of Generative AI](https://mlsanity.com/paper/2406.07457/) (2024)

## Built on by (most cited first, 2 of 2)

- [AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration](https://mlsanity.com/paper/2604.01014/) (2026)
- [Future Anchored Verification and Online Recovery for World Action Models](https://mlsanity.com/paper/2610.06280/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements - posing a critical barrier to their trustworthy deployment. In this work, we propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition. Our framework is grounded in a normative principle of computational rationality, allowing us to model a VLM's generation as a dynamic cognitive trajectory. We design a suite of information-theoretic probes that project this trajectory onto an interpretable, low-dimensional Cognitive State Space. Our central discovery is a governing principle we term the geometric-information duality: a cognitive trajectory's geometric abnormality within this space is fundamentally equivalent to its high information-theoretic surprisal. Hallucination detection is counts as a geometric anomaly detection problem. Evaluated across diverse settings - from rigorous binary QA (POPE) and comprehensive reasoning (MME) to unconstrained open-ended captioning (MS-COCO) - our framework achieves state-of-the-art performance. Crucially, it operates with high efficiency under weak supervision and remains highly robust even when calibration data is heavily contaminated. This approach enables a causal attribution of failures, mapping observable errors to distinct pathological states: perceptual instability (measured by Perceptual Entropy), logical-causal failure (measured by Inferential Conflict), and decisional ambiguity (measured by Decision Entropy). Ultimately, this opens a path toward building AI systems whose reasoning is transparent, auditable, and diagnosable by design.