mlsanityData to 14 Oct 2026

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

arXiv 2111.09525 ↗First posted 18 Nov 2021cs.CLcited by 407

Not on the map. Why?

Cited by 27 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. On our newly introduced benchmark called SummaC (Summary Consistency) consisting of six large inconsistency detection datasets, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% point improvement compared to prior work. We make the models and datasets available: https://github.com/tingofurro/summac

Key facts. Cited by 407 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

30 of 41 references.

References · 30

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  12. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
18 more
  1. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  2. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  3. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  4. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  5. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  6. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  7. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  8. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  9. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  10. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  11. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  12. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  13. 2020Multi-Fact Correction in Abstractive Text Summarizationcited by 66
  14. 2020What Have We Achieved on Text Summarization?cited by 65
  15. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  16. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  17. 2019Guiding Extractive Summarization with Question-Answering Rewardscited by 13
  18. 2021SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarizationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 36; 2023: 80; 2024: 134; 2025: 67; 2026: 89.

407 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  10. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  11. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  12. 2023Large Language Model Alignment: A Surveycited by 243
38 more
  1. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  2. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  3. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  4. 2023Conformal Language Modelingcited by 170
  5. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  6. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  7. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  8. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  9. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  10. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  11. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 99
  12. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  13. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  14. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  15. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  16. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  17. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  18. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  19. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  20. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  21. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  22. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  23. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  24. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  25. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 50
  26. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
  27. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  28. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  29. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  30. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  31. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  32. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  33. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  34. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  35. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  36. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  37. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 37
  38. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37

50 most cited of 407 All 407 (JSON)