# Reference-free Hallucination Detection for Large Vision-Language Models

arXiv: https://arxiv.org/abs/2408.05767 · HTML: https://mlsanity.com/paper/2408.05767/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 11 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 11 of 11)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States](https://mlsanity.com/paper/2402.09733/) (2024)
- [FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2311.01477/) (2023)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models](https://mlsanity.com/paper/2307.10236/) (2023)
- [Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers](https://mlsanity.com/paper/2312.04333/) (2023)
- [Multimodal Large Language Models to Support Real-World Fact-Checking](https://mlsanity.com/paper/2403.03627/) (2024)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models](https://mlsanity.com/paper/2503.01854/) (2025)
- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2503.14530/) (2025)
- [TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention](https://mlsanity.com/paper/2503.10602/) (2025)
- [GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity](https://mlsanity.com/paper/2508.19972/) (2025)
- [Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume](https://mlsanity.com/paper/2602.24195/) (2026)
- [UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Models](https://mlsanity.com/paper/2503.20504/) (2025)
- [SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2502.01812/) (2025)
- [When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQA](https://mlsanity.com/paper/2511.01458/) (2025)
- [Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering](https://mlsanity.com/paper/2511.19768/) (2025)
- [V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering](https://mlsanity.com/paper/2601.18240/) (2026)
- [Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models](https://mlsanity.com/paper/2606.15910/) (2026)
- [PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models](https://mlsanity.com/paper/2511.11502/) (2025)
- [Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2603.16558/) (2026)
- [Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gain](https://mlsanity.com/paper/2603.21693/) (2026)
- [SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems](https://mlsanity.com/paper/2603.23853/) (2026)
- [Hallucination-aware intermediate representation edit in large vision-language models](https://mlsanity.com/paper/2603.29405/) (2026)
- [A Benchmark for Hallucination Detection in VLMs for Gastrointestinal Endoscopy](https://mlsanity.com/paper/2606.24115/) (2026)
- [Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?](https://mlsanity.com/paper/2606.31407/) (2026)
- [UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space](https://mlsanity.com/paper/2608.03817/) (2026)
- [How Much Do Legal RAG Systems Still Hallucinate?](https://mlsanity.com/paper/2608.14210/) (2026)
- [CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning](https://mlsanity.com/paper/2609.05708/) (2026)
- [From Routing Signals to Selective Review: Visual regrounding in MoE VLMs](https://mlsanity.com/paper/2609.38111/) (2026)

## Abstract (arXiv)

Large vision-language models (LVLMs) have made significant progress in recent years. While LVLMs exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, they are prone to producing hallucinations. While several methods are proposed to evaluate the hallucinations in LVLMs, most are reference-based and depend on external tools, which complicates their practical application. To assess the viability of alternative methods, it is critical to understand whether the reference-free approaches, which do not rely on any external tools, can efficiently detect hallucinations. Therefore, we initiate an exploratory study to demonstrate the effectiveness of different reference-free solutions in detecting hallucinations in LVLMs. In particular, we conduct an extensive study on three kinds of techniques: uncertainty-based, consistency-based, and supervised uncertainty quantification methods on four representative LVLMs across two different tasks. The empirical results show that the reference-free approaches are capable of effectively detecting non-factual responses in LVLMs, with the supervised uncertainty quantification method outperforming the others, achieving the best performance across different settings.