mlsanityData to 7 Oct 2026

Evaluating Factuality in Generation with Dependency-level Entailment

arXiv 2010.05478 ↗First posted 12 Oct 2020cs.CLcited by 122

Not on the map. Why?

Abstract (arXiv)

Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 29; 2023: 27; 2024: 34; 2025: 7; 2026: 11.

122 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  8. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  9. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  11. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  12. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
38 more
  1. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  2. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  3. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  4. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  5. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  6. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  7. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  8. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  9. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  10. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  11. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  12. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  13. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  14. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  15. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  16. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  17. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  18. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  19. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  20. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  21. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  22. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 49
  23. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  24. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  25. 2022FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representationscited by 41
  26. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  27. 2024A Closer Look at Claim Decompositioncited by 40
  28. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  29. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  30. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 37
  31. 2022Evaluating Factuality in Text Simplificationcited by 34
  32. 2023Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinationscited by 34
  33. 2022Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarizationcited by 32
  34. 2022PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognitioncited by 29
  35. 2022Learning to Revise References for Faithful Summarizationcited by 28
  36. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  37. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  38. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26

50 most cited of 122 All 122 (JSON)