# Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters

arXiv: https://arxiv.org/abs/2204.07447 · HTML: https://mlsanity.com/paper/2204.07447/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 75 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [Towards Debiasing Fact Verification Models](https://mlsanity.com/paper/1908.05267/) (2019)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [End-to-End Bias Mitigation by Modelling Biases in Corpora](https://mlsanity.com/paper/1909.06321/) (2019)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)

## Built on by (most cited first, 30 of 44)

- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://mlsanity.com/paper/2309.07852/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models](https://mlsanity.com/paper/2305.11171/) (2023)
- [Understanding Retrieval Augmentation for Long-Form Question Answering](https://mlsanity.com/paper/2310.12150/) (2023)
- [mFACE: Multilingual Summarization with Factual Consistency Evaluation](https://mlsanity.com/paper/2212.10622/) (2022)
- [PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition](https://mlsanity.com/paper/2212.10750/) (2022)
- [Fast and Accurate Factual Inconsistency Detection Over Long Documents](https://mlsanity.com/paper/2310.13189/) (2023)
- [Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks](https://mlsanity.com/paper/2402.17630/) (2024)
- [Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations](https://mlsanity.com/paper/2311.04335/) (2023)
- [Atomic Inference for NLI with Generated Facts as Atoms](https://mlsanity.com/paper/2305.13214/) (2023)
- [ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search](https://mlsanity.com/paper/2509.23519/) (2025)
- [The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models](https://mlsanity.com/paper/2404.05904/) (2024)
- [FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document](https://mlsanity.com/paper/2404.11184/) (2024)
- [Revisiting text decomposition methods for NLI-based factuality scoring of summaries](https://mlsanity.com/paper/2211.16853/) (2022)
- [InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification](https://mlsanity.com/paper/2401.16475/) (2024)
- [Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systems](https://mlsanity.com/paper/2209.03755/) (2022)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models](https://mlsanity.com/paper/2311.09182/) (2023)
- [Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports](https://mlsanity.com/paper/2304.13180/) (2023)
- [Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization](https://mlsanity.com/paper/2502.06185/) (2025)
- [μPLAN: Summarizing using a Content Plan as Cross-Lingual Bridge](https://mlsanity.com/paper/2305.14205/) (2023)
- [LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction](https://mlsanity.com/paper/2305.19585/) (2023)
- [LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval](https://mlsanity.com/paper/2212.03222/) (2022)
- [Drafting Event Schemas using Language Models](https://mlsanity.com/paper/2305.14847/) (2023)
- [Minimal Evidence Group Identification for Claim Verification](https://mlsanity.com/paper/2404.15588/) (2024)
- [SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes](https://mlsanity.com/paper/2310.18431/) (2023)
- [Explicating the Implicit: Argument Detection Beyond Sentence Boundaries](https://mlsanity.com/paper/2408.04246/) (2024)
- [Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models](https://mlsanity.com/paper/2509.23233/) (2025)

## Abstract (arXiv)

Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. While early work identified certain biases in NLI models, recent advancements in modeling and datasets demonstrated promising performance. In this work, we further explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. First, we analyze the robustness of these models to longer and out-of-domain inputs. Then, we develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset. Interestingly, we find NLI scores to provide strong retrieval signals, leading to more relevant evidence extractions compared to common similarity-based methods. Finally, we go further and investigate whole document clusters to identify both discrepancies and consensus among sources. In a test case, we find real inconsistencies between Wikipedia pages in different languages about the same topic.