# Chainpoll: A high efficacy method for LLM hallucination detection

arXiv: https://arxiv.org/abs/2310.18344 · HTML: https://mlsanity.com/paper/2310.18344/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://mlsanity.com/paper/2210.02627/) (2022)
- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [Evaluating Factual Consistency of Summaries with Large Language Models](https://mlsanity.com/paper/2305.14069/) (2023)

## Built on by (most cited first, 30 of 39)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability](https://mlsanity.com/paper/2410.11414/) (2024)
- [Multicalibration for Confidence Scoring in LLMs](https://mlsanity.com/paper/2404.04689/) (2024)
- [Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective](https://mlsanity.com/paper/2505.12886/) (2025)
- [Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning](https://mlsanity.com/paper/2410.16130/) (2024)
- [DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applications](https://mlsanity.com/paper/2409.19020/) (2024)
- [Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples](https://mlsanity.com/paper/2505.14518/) (2025)
- [The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States](https://mlsanity.com/paper/2412.17056/) (2024)
- [Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling](https://mlsanity.com/paper/2409.11283/) (2024)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders](https://mlsanity.com/paper/2512.08892/) (2025)
- [Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis](https://mlsanity.com/paper/2502.08663/) (2025)
- [Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation](https://mlsanity.com/paper/2507.00057/) (2025)
- [Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence](https://mlsanity.com/paper/2509.23573/) (2025)
- [HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders](https://mlsanity.com/paper/2604.16430/) (2026)
- [LightHouse: A Survey of AGI Hallucination](https://mlsanity.com/paper/2401.06792/) (2024)
- [SafeLLM: Domain-Specific Safety Monitoring for Large Language Models: A Case Study of Offshore Wind Maintenance](https://mlsanity.com/paper/2410.10852/) (2024)
- [FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verification](https://mlsanity.com/paper/2604.23588/) (2026)
- [Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System](https://mlsanity.com/paper/2505.20310/) (2025)
- [KEA Explain: Explanations of Hallucinations using Graph Kernel Analysis](https://mlsanity.com/paper/2507.03847/) (2025)
- [A Debate-Driven Experiment on LLM Hallucinations and Accuracy](https://mlsanity.com/paper/2410.19485/) (2024)
- [Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledge](https://mlsanity.com/paper/2411.09689/) (2024)
- [TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG](https://mlsanity.com/paper/2512.07515/) (2025)
- [INTRYGUE: Induction-Aware Entropy Gating for Reliable RAG Uncertainty Estimation](https://mlsanity.com/paper/2603.21607/) (2026)
- [SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection](https://mlsanity.com/paper/2404.03732/) (2024)
- [RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation](https://mlsanity.com/paper/2411.03920/) (2024)
- [PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models](https://mlsanity.com/paper/2411.15320/) (2024)
- [NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPT](https://mlsanity.com/paper/2503.01921/) (2025)
- [Towards a rigorous evaluation of RAG systems: the challenge of due diligence](https://mlsanity.com/paper/2507.21753/) (2025)
- [Error Detection and Correction for Interpretable Mathematics in Large Language Models](https://mlsanity.com/paper/2508.03500/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have experienced notable advancements in generating coherent and contextually relevant responses. However, hallucinations - incorrect or unfounded claims - are still prevalent, prompting the creation of automated metrics to detect these in LLM outputs. Our contributions include: introducing ChainPoll, an innovative hallucination detection method that excels compared to its counterparts, and unveiling RealHall, a refined collection of benchmark datasets to assess hallucination detection metrics from recent studies. While creating RealHall, we assessed tasks and datasets from previous hallucination detection studies and observed that many are not suitable for the potent LLMs currently in use. Overcoming this, we opted for four datasets challenging for modern LLMs and pertinent to real-world scenarios. Using RealHall, we conducted a comprehensive comparison of ChainPoll with numerous hallucination metrics from recent studies. Our findings indicate that ChainPoll outperforms in all RealHall benchmarks, achieving an overall AUROC of 0.781. This surpasses the next best theoretical method by 11% and exceeds industry standards by over 23%. Additionally, ChainPoll is cost-effective and offers greater transparency than other metrics. We introduce two novel metrics to assess LLM hallucinations: Adherence and Correctness. Adherence is relevant to Retrieval Augmented Generation workflows, evaluating an LLM's analytical capabilities within given documents and contexts. In contrast, Correctness identifies logical and reasoning errors.