# LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval arXiv: https://arxiv.org/abs/2212.03222 ยท HTML: https://mlsanity.com/paper/2212.03222/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 57 references that are in the map. - First posted on arXiv 6 Dec 2022. ## Builds on (most cited first, 30 of 31) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017) - [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017) - [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016) - [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020) - [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020) - [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020) - [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018) - [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021) - [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018) - [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020) - [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021) - [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021) - [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018) - [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022) - [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021) - [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018) - [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021) - [Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment](https://mlsanity.com/paper/1804.08117/) (2018) - [The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes](https://mlsanity.com/paper/2012.14210/) (2020) - [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022) - [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022) - [Evaluating Gender Bias in Natural Language Inference](https://mlsanity.com/paper/2105.05541/) (2021) - [Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts](https://mlsanity.com/paper/2011.04864/) (2020) ## Built on by (most cited first, 6 of 6) - [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024) - [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024) - [Improve Dense Passage Retrieval with Entailment Tuning](https://mlsanity.com/paper/2410.15801/) (2024) - [LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2410.12064/) (2024) - [ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts](https://mlsanity.com/paper/2605.00116/) (2026) - [Augmenting Legal Decision Support Systems with LLM-based NLI for Analyzing Social Media Evidence](https://mlsanity.com/paper/2410.15990/) (2024) ## Abstract (arXiv) Natural language inference has trended toward studying contexts beyond the sentence level. An important application area is law: past cases often do not foretell how they apply to new situations and implications must be inferred. This paper introduces LawngNLI, constructed from U.S. legal opinions with automatic labels with high human-validated accuracy. Premises are long and multigranular. Experiments show two use cases. First, LawngNLI can benchmark for in-domain generalization from short to long contexts. It has remained unclear if large-scale long-premise NLI datasets actually need to be constructed: near-top performance on long premises could be achievable by fine-tuning using short premises. Without multigranularity, benchmarks cannot distinguish lack of fine-tuning on long premises versus domain shift between short and long datasets. In contrast, our long and short premises share the same examples and domain. Models fine-tuned using several past NLI datasets and/or our short premises fall short of top performance on our long premises. So for at least certain domains (such as ours), large-scale long-premise datasets are needed. Second, LawngNLI can benchmark for implication-based retrieval. Queries are entailed or contradicted by target documents, allowing users to move between arguments and evidence. Leading retrieval models perform reasonably zero shot on a LawngNLI-derived retrieval task. We compare different systems for re-ranking, including lexical overlap and cross-encoders fine-tuned using a modified LawngNLI or past NLI datasets. LawngNLI can train and test systems for implication-based case retrieval and argumentation.