mlsanityData to 14 Oct 2026

FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs

arXiv 2410.13210 ↗First posted 17 Oct 2024cs.CL cs.AIcited by 24

On the map: Retrieval-augmented generation → Hallucination detection in RAG

Abstract (arXiv)

Summarization is one of the most common tasks performed by large language models (LLMs), especially in applications like Retrieval-Augmented Generation (RAG). However, existing evaluations of hallucinations in LLM-generated summaries, and evaluations of hallucination detection models both suffer from a lack of diversity and recency in the LLM and LLM families considered. This paper introduces FaithBench, a summarization hallucination benchmark comprising challenging hallucinations made by 10 modern LLMs from 8 different families, with ground truth annotations by human experts. ``Challenging'' here means summaries on which popular, state-of-the-art hallucination detection models, including GPT-4o-as-a-judge, disagreed on. Our results show GPT-4o and GPT-3.5-Turbo produce the least hallucinations. However, even the best hallucination detection models have near 50% accuracies on FaithBench, indicating lots of room for future improvement. The repo is https://github.com/vectara/FaithBench

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection in RAG, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 35 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

21 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models31 Dec

Other references · 20

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  8. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  10. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  11. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
  12. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
8 more
  1. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  2. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  3. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  4. 2024GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Frameworkcited by 30
  5. 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
  6. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 27
  7. 2024Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trendscited by 25
  8. 2024SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakescited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 13.

24 papers.

Sort:

By area:

  1. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  2. 2026HalluHard: A Hard Multi-Turn Hallucination Benchmarkcited by 9
  3. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Modelscited by 2
  4. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  5. 2025HAD: HAllucination Detection Language Models Based on a Comprehensive Hallucination Taxonomycited by 1
  6. 2025Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacycited by 1
  7. 2026A Geometric Taxonomy of Hallucinations in LLMscited by 1
  8. 2026BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengalicited by 1
  9. 2025MIH-TCCT: Mitigating Inconsistent Hallucinations in LLMs via Event-Driven Text-Code Cyclic Trainingcited by 0
  10. 2025UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Outputcited by 0
  11. 2025MMD-Flagger: Leveraging Maximum Mean Discrepancy to Detect Hallucinationscited by 0
  12. 2025Machine Mirages: Defining the Undefinedcited by 0
12 more
  1. 2025FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcriptscited by 0
  2. 2025SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generationcited by 0
  3. 2025Detecting Hallucinations in Authentic LLM-Human Interactionscited by 0
  4. 2025Semantic Grounding Index: Geometric Bounds on Context Engagement in RAG Systemscited by 0
  5. 2026LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMscited by 0
  6. 2026Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agentscited by 0
  7. 2026Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAGcited by 0
  8. 2026Averaging Bias: Human Faithfulness Annotations are not Locally Faithfulcited by 0
  9. 2026Decoupling Generation and Selection for Budget-Constrained Faithful Summarizationcited by 0
  10. 2026HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviewscited by 0
  11. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0
  12. 2026Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Don't Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models
  2. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
  3. 2026Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
  4. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards
  5. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  6. 2024THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
  8. 2025Bi'an: A Bilingual Benchmark and Model for Hallucination Detection in Retrieval-Augmented Generation