mlsanityData to 7 Oct 2026

Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection

arXiv 2502.15845 ↗First posted 20 Feb 2025cs.CL cs.AIcited by 18

On the map: LLM reasoning → Hallucination detection and mitigation

Abstract (arXiv)

Large Language Models (LLMs) often hallucinate, limiting their reliability in sensitive applications. In black-box settings, several self-consistency-based techniques have been proposed for hallucination detection. We empirically show that these methods perform nearly as well as a supervised (black-box) oracle, leaving limited room for further gains within this paradigm. To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM. With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods. We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases. It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier. We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights. Extensive experiments on QA-style hallucination detection benchmarks show that this approach maintains high detection performance while significantly reducing computational cost.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

25 of 29 references.

References · 25

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
13 more
  1. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  2. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  3. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  4. 2023Fine-tuning Language Models for Factualitycited by 219
  5. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  6. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  7. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  8. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  9. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  10. 2023Do Language Models Know When They're Hallucinating References?cited by 115
  11. 2023Calibrated Language Models Must Hallucinatecited by 114
  12. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  13. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 12.

18 papers.

Sort:

By area:

  1. 2025Why Language Models Hallucinatecited by 205
  2. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  3. 2025Too Consistent to Detect: A Study of Self-Consistent Errors in LLMscited by 18
  4. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  5. 2026HALT: Hallucination Assessment via Log-probs as Time seriescited by 8
  6. 2026Cross-Model Disagreement as a Label-Free Correctness Signalcited by 4
  7. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  8. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 3
  9. 2025Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMscited by 3
  10. 2026Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimationcited by 3
  11. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  12. 2026Two Pathways to Truthfulness: On the Intrinsic Encoding of LLM Hallucinationscited by 1
6 more
  1. 2026EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectorscited by 1
  2. 2026Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failurescited by 1
  3. 2026SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systemscited by 0
  4. 2026Mind the Unseen Mass: Unmasking LLM Hallucinations via Soft-Hybrid Alphabet Estimationcited by 0
  5. 2026Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurementcited by 0
  6. 2026NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Passcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BEACON: Behavioral Entropy Aggregation for Cross-Model Hallucination Detection in Large Language Models
  2. 2025SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models
  3. 2025Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection
  4. 2026Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments
  5. 2026UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
  6. 2025D2HScore: Reasoning-Aware Hallucination Detection via Semantic Breadth and Depth Analysis in LLMs
  7. 2026AURORA: Asymmetry and Update-Induced Rotation for Robust Hallucination Detection in Large Language Models
  8. 2025Revisiting Hallucination Detection with Effective Rank-based Uncertainty