mlsanityData to 14 Oct 2026

Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization

arXiv 2502.06185 ↗First posted 10 Feb 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Detecting factual inconsistency for long document summarization remains challenging, given the complex structure of the source article and long summary length. In this work, we study factual inconsistency errors and connect them with a line of discourse analysis. We find that errors are more common in complex sentences and are associated with several discourse features. We propose a framework that decomposes long texts into discourse-inspired chunks and utilizes discourse information to better aggregate sentence-level scores predicted by natural language inference models. Our approach shows improved performance on top of different model baselines over several evaluation benchmarks, covering rich domains of texts, focusing on long document summarization. This underscores the significance of incorporating discourse features in developing models for scoring summaries for long document factual inconsistency.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

45 of 65 references.

References · 45

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  6. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  7. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  8. 2021Efficient Attentions for Long Document Summarizationcited by 345
  9. 2023How Language Model Hallucinations Can Snowballcited by 311
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  12. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
33 more
  1. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  2. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  3. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  4. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  5. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  6. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  7. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  8. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 105
  9. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  10. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  11. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  12. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  13. 2021DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsingcited by 28
  14. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  15. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  16. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 27
  17. 2022ArgLegalSumm: Improving Abstractive Summarization of Legal Documents with Argument Miningcited by 25
  18. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  19. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  20. 2019Discourse Level Factors for Sentence Deletion in Text Simplificationcited by 19
  21. 2024Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifscited by 18
  22. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  23. 2023LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisationcited by 16
  24. 2024FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Documentcited by 15
  25. 2022Revisiting text decomposition methods for NLI-based factuality scoring of summariescited by 14
  26. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  27. 2023Generating EDU Extracts for Plan-Guided Summary Re-Rankingcited by 9
  28. 2023Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Rerankingcited by 9
  29. 2023What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarizationcited by 8
  30. 2017Discourse Structure in Machine Translation Evaluationcited by 7
  31. 2024Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherencecited by 7
  32. 2017Sentiment analysis based on rhetorical structure theory: Learning deep neural networks from discourse treescited by 6
  33. 2021W-RST: Towards a Weighted RST-style Discourse Frameworkcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMscited by 7
  2. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  3. 2025Stress Testing Factual Consistency Metrics for Long-Document Summarizationcited by 2
  4. 2026Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarizationcited by 0
  5. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
  6. 2026A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarizationcited by 0
  7. 2026Averaging Bias: Human Faithfulness Annotations are not Locally Faithfulcited by 0
  8. 2026Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillationcited by 0