# Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach

arXiv: https://arxiv.org/abs/2405.19648 · HTML: https://mlsanity.com/paper/2405.19648/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 28 of 28)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models](https://mlsanity.com/paper/2403.06448/) (2024)
- [Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinations](https://mlsanity.com/paper/2310.03951/) (2023)
- [Diving Deep into Modes of Fact Hallucinations in Dialogue Systems](https://mlsanity.com/paper/2301.04449/) (2023)
- [Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models](https://mlsanity.com/paper/2310.07653/) (2023)

## Built on by (most cited first, 30 of 42)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation](https://mlsanity.com/paper/2506.17088/) (2025)
- [Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs](https://mlsanity.com/paper/2502.21239/) (2025)
- [Robust Hallucination Detection in LLMs via Adaptive Token Selection](https://mlsanity.com/paper/2504.07863/) (2025)
- [Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs](https://mlsanity.com/paper/2509.13813/) (2025)
- [Quantized Delta Weight Is Safety Keeper](https://mlsanity.com/paper/2411.19530/) (2024)
- [Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate](https://mlsanity.com/paper/2509.04492/) (2025)
- [FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formats](https://mlsanity.com/paper/2408.08841/) (2024)
- [Prompt-Guided Internal States for Hallucination Detection of Large Language Models](https://mlsanity.com/paper/2411.04847/) (2024)
- [HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMs](https://mlsanity.com/paper/2504.09482/) (2025)
- [HALT: Hallucination Assessment via Log-probs as Time series](https://mlsanity.com/paper/2602.02888/) (2026)
- [Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Method](https://mlsanity.com/paper/2601.20026/) (2026)
- [Span-Level Hallucination Detection for LLM-Generated Answers](https://mlsanity.com/paper/2504.18639/) (2025)
- [MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probing](https://mlsanity.com/paper/2605.24919/) (2026)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [First Hallucination Tokens Are Different from Conditional Ones](https://mlsanity.com/paper/2507.20836/) (2025)
- [Addressing Hallucinations with RAG and NMISS in Italian Healthcare LLM Chatbots](https://mlsanity.com/paper/2412.04235/) (2024)
- [Entropy-Tree: Tree-Based Decoding with Entropy-Guided Exploration](https://mlsanity.com/paper/2601.15296/) (2026)
- [Low-Cost Black-Box Detection of LLM Hallucinations via Dynamical System Prediction](https://mlsanity.com/paper/2605.05134/) (2026)
- [Developing a Reliable, Fast, General-Purpose Hallucination Detection and Mitigation Service](https://mlsanity.com/paper/2407.15441/) (2024)
- [Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledge](https://mlsanity.com/paper/2411.09689/) (2024)
- [What are Models Thinking about? Understanding Large Language Model Hallucinations "Psychology" through Model Inner State Analysis](https://mlsanity.com/paper/2502.13490/) (2025)
- [From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models](https://mlsanity.com/paper/2505.24232/) (2025)
- [PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length](https://mlsanity.com/paper/2602.01274/) (2026)
- [Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor](https://mlsanity.com/paper/2507.15903/) (2025)
- [From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors](https://mlsanity.com/paper/2509.20065/) (2025)
- [Leveraging NTPs for Efficient Hallucination Detection in VLMs](https://mlsanity.com/paper/2509.20379/) (2025)
- [HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing](https://mlsanity.com/paper/2509.23835/) (2025)
- [Teaming LLMs to Detect and Mitigate Hallucinations](https://mlsanity.com/paper/2510.19507/) (2025)
- [FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answering](https://mlsanity.com/paper/2601.00269/) (2026)

## Abstract (arXiv)

Concerns regarding the propensity of Large Language Models (LLMs) to produce inaccurate outputs, also known as hallucinations, have escalated. Detecting them is vital for ensuring the reliability of applications relying on LLM-generated content. Current methods often demand substantial resources and rely on extensive LLMs or employ supervised learning with multidimensional features or intricate linguistic and semantic analyses difficult to reproduce and largely depend on using the same LLM that hallucinated. This paper introduces a supervised learning approach employing two simple classifiers utilizing only four numerical features derived from tokens and vocabulary probabilities obtained from other LLM evaluators, which are not necessarily the same. The method yields promising results, surpassing state-of-the-art outcomes in multiple tasks across three different benchmarks. Additionally, we provide a comprehensive examination of the strengths and weaknesses of our approach, highlighting the significance of the features utilized and the LLM employed as an evaluator. We have released our code publicly at https://github.com/Baylor-AI/HalluDetect.