# Estimating the Hallucination Rate of Generative AI

arXiv: https://arxiv.org/abs/2406.07457 · HTML: https://mlsanity.com/paper/2406.07457/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 101 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 30 of 69)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)

## Built on by (most cited first, 20 of 20)

- [Estimating Knowledge in Large Language Models Without Generating a Single Token](https://mlsanity.com/paper/2406.12673/) (2024)
- [RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration](https://mlsanity.com/paper/2503.13514/) (2025)
- [Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems](https://mlsanity.com/paper/2506.10060/) (2025)
- [Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation](https://mlsanity.com/paper/2505.17099/) (2025)
- [Do Large Language Models (Really) Need Statistical Foundations?](https://mlsanity.com/paper/2505.19145/) (2025)
- [LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs](https://mlsanity.com/paper/2506.11148/) (2025)
- [Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings](https://mlsanity.com/paper/2508.06030/) (2025)
- [Can Generative AI Solve Your In-Context Learning Problem? A Martingale Perspective](https://mlsanity.com/paper/2412.06033/) (2024)
- [Balancing Content Size in RAG-Text2SQL System](https://mlsanity.com/paper/2502.15723/) (2025)
- [Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2511.17946/) (2025)
- [Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers](https://mlsanity.com/paper/2512.10422/) (2025)
- [Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models](https://mlsanity.com/paper/2603.15557/) (2026)
- [Rejecting Hallucinated State Targets during Planning](https://mlsanity.com/paper/2410.07096/) (2024)
- [CHEM: Estimating and Understanding Hallucinations in Deep Learning for Image Processing](https://mlsanity.com/paper/2512.09806/) (2025)
- [What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness](https://mlsanity.com/paper/2605.30911/) (2026)
- [Machine Mirages: Defining the Undefined](https://mlsanity.com/paper/2506.13990/) (2025)
- [In-Context Function Learning in Large Language Models](https://mlsanity.com/paper/2602.11863/) (2026)
- [Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering](https://mlsanity.com/paper/2605.01827/) (2026)
- [TIGER: Text-Informed Generalized Enzyme-Reaction Retrieval](https://mlsanity.com/paper/2605.24489/) (2026)
- [The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining](https://mlsanity.com/paper/2609.32964/) (2026)

## Abstract (arXiv)

This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a hallucination as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.