# The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models

arXiv: https://arxiv.org/abs/2404.05904 · HTML: https://mlsanity.com/paper/2404.05904/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)

## Built on by (most cited first, 17 of 17)

- [Logically Consistent Language Models via Neuro-Symbolic Integration](https://mlsanity.com/paper/2409.13724/) (2024)
- [DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations](https://mlsanity.com/paper/2410.18860/) (2024)
- [Conformal Linguistic Calibration: Trading-off between Factuality and Specificity](https://mlsanity.com/paper/2502.19110/) (2025)
- [Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information](https://mlsanity.com/paper/2506.11680/) (2025)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2505.11415/) (2025)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [Self-Training Large Language Models for Tool-Use Without Demonstrations](https://mlsanity.com/paper/2502.05867/) (2025)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts](https://mlsanity.com/paper/2508.06361/) (2025)
- [SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2502.01812/) (2025)
- [ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Models](https://mlsanity.com/paper/2411.10867/) (2024)
- [An Analysis of Decoding Methods for LLM-based Agents for Faithful Multi-Hop Question Answering](https://mlsanity.com/paper/2503.23415/) (2025)
- [Neural Diversity Regularizes Hallucinations in Language Models](https://mlsanity.com/paper/2510.20690/) (2025)
- [Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity](https://mlsanity.com/paper/2602.00723/) (2026)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2412.07067/) (2024)
- [Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights](https://mlsanity.com/paper/2605.11330/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text. However, these models are prone to ``hallucinations'' -- outputs that do not align with factual reality or the input context. This paper introduces the Hallucinations Leaderboard, an open initiative to quantitatively measure and compare the tendency of each model to produce hallucinations. The leaderboard uses a comprehensive set of benchmarks focusing on different aspects of hallucinations, such as factuality and faithfulness, across various tasks, including question-answering, summarisation, and reading comprehension. Our analysis provides insights into the performance of different models, guiding researchers and practitioners in choosing the most reliable models for their applications.