# Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection

arXiv: https://arxiv.org/abs/2502.15845 · HTML: https://mlsanity.com/paper/2502.15845/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- LLM reasoning / [Hallucination detection and mitigation](https://mlsanity.com/lr/hallucination-detection-and-mitigation/)

## Builds on (most cited first, 25 of 25)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [Calibrated Language Models Must Hallucinate](https://mlsanity.com/paper/2311.14648/) (2023)
- [SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency](https://mlsanity.com/paper/2311.01740/) (2023)
- [Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension](https://mlsanity.com/paper/2402.18048/) (2024)

## Built on by (most cited first, 18 of 18)

- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs](https://mlsanity.com/paper/2505.17656/) (2025)
- [Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification](https://mlsanity.com/paper/2604.17112/) (2026)
- [HALT: Hallucination Assessment via Log-probs as Time series](https://mlsanity.com/paper/2602.02888/) (2026)
- [Cross-Model Disagreement as a Label-Free Correctness Signal](https://mlsanity.com/paper/2603.25450/) (2026)
- [Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation](https://mlsanity.com/paper/2511.11500/) (2025)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMs](https://mlsanity.com/paper/2512.23547/) (2025)
- [Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation](https://mlsanity.com/paper/2603.05881/) (2026)
- [A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models](https://mlsanity.com/paper/2606.19868/) (2026)
- [Two Pathways to Truthfulness: On the Intrinsic Encoding of LLM Hallucinations](https://mlsanity.com/paper/2601.07422/) (2026)
- [EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors](https://mlsanity.com/paper/2604.02784/) (2026)
- [Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures](https://mlsanity.com/paper/2609.10969/) (2026)
- [SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems](https://mlsanity.com/paper/2603.23853/) (2026)
- [Mind the Unseen Mass: Unmasking LLM Hallucinations via Soft-Hybrid Alphabet Estimation](https://mlsanity.com/paper/2604.19162/) (2026)
- [Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement](https://mlsanity.com/paper/2605.05103/) (2026)
- [NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Pass](https://mlsanity.com/paper/2609.14448/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) often hallucinate, limiting their reliability in sensitive applications. In black-box settings, several self-consistency-based techniques have been proposed for hallucination detection. We empirically show that these methods perform nearly as well as a supervised (black-box) oracle, leaving limited room for further gains within this paradigm. To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM. With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods. We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases. It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier. We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights. Extensive experiments on QA-style hallucination detection benchmarks show that this approach maintains high detection performance while significantly reducing computational cost.