mlsanityData to 7 Oct 2026

TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization

arXiv 2402.13249 ↗First posted 20 Feb 2024cs.CL cs.AIcited by 66

Not on the map. Why?

Abstract (arXiv)

Single document news summarization has seen substantial progress on faithfulness in recent years, driven by research on the evaluation of factual consistency, or hallucinations. We ask whether these advances carry over to other text summarization domains. We propose a new evaluation benchmark on topic-focused dialogue summarization, generated by LLMs of varying sizes. We provide binary sentence-level human annotations of the factual consistency of these summaries along with detailed explanations of factually inconsistent sentences. Our analysis shows that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model's size. On the other hand, when LLMs, including GPT-4, serve as binary factual evaluators, they perform poorly and can be outperformed by prevailing state-of-the-art specialized factuality evaluation metrics. Finally, we conducted an analysis of hallucination types with a curated error taxonomy. We find that there are diverse errors and error distributions in model-generated summaries and that non-LLM based metrics can capture all error types better than LLM-based evaluators.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

22 of 41 references.

References · 22

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  7. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
10 more
  1. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  2. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  3. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  4. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  5. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  6. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  7. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  8. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  9. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  10. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 28; 2026: 14.

66 papers.

Sort:

By area:

  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  2. 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 75
  3. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  4. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 35
  5. 2024A Survey of Useful LLM Evaluationcited by 28
  6. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  7. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  8. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  9. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  10. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  11. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 19
  12. 2024FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Documentcited by 15
38 more
  1. 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Datacited by 14
  2. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
  3. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  4. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languagescited by 10
  5. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9
  6. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 9
  7. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaborationcited by 9
  8. 2025CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Modelscited by 9
  9. 2024Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasiscited by 8
  10. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metricscited by 8
  11. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  12. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generationcited by 7
  13. 2025Multi-Party Conversational Agents: A Surveycited by 6
  14. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarizationcited by 6
  15. 2025Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoderscited by 6
  16. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  17. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 5
  18. 2024Localizing and Mitigating Errors in Long-form Question Answeringcited by 4
  19. 2024Learning to Verify Summary Facts with Fine-Grained LLM Feedbackcited by 4
  20. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4
  21. 2025Long-Form Information Alignment Evaluation Beyond Atomic Factscited by 4
  22. 2024Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization?cited by 3
  23. 2024Do Large Language Models Possess Sensitive to Sentiment?cited by 3
  24. 2024Differentially Private Learning Needs Better Model Initialization and Self-Distillationcited by 3
  25. 2025Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systemscited by 3
  26. 2025FaithLens: Detecting and Explaining Faithfulness Hallucinationcited by 3
  27. 2024Detecting Errors through Ensembling Prompts (DEEP): An End-to-End LLM Framework for Detecting Factual Errorscited by 2
  28. 2024LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reportscited by 2
  29. 2025GPT Editors, Not Authors: The Stylistic Footprint of LLMs in Academic Preprintscited by 2
  30. 2025Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contextscited by 2
  31. 2025Controllable Abstraction in Summary Generation for Large Language Models via Prompt Engineeringcited by 2
  32. 2024SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Editscited by 1
  33. 2025Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Datacited by 1
  34. 2025OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matchingcited by 1
  35. 2025Multilingual Self-Taught Faithfulness Evaluatorscited by 1
  36. 2025Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarizationcited by 1
  37. 2025DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queriescited by 1
  38. 2026InFi-Check: Interpretable and Fine-Grained Fact-Checking of LLMscited by 1

50 most cited of 66 All 66 (JSON)