# Non-entailed subsequences as a challenge for natural language inference

arXiv: https://arxiv.org/abs/1811.12112 · HTML: https://mlsanity.com/paper/1811.12112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2018.

## Built on by (most cited first, 5 of 5)

- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning](https://mlsanity.com/paper/2305.19426/) (2023)
- [Estimating the Causal Effects of Natural Logic Features in Neural NLI Models](https://mlsanity.com/paper/2305.08572/) (2023)
- [ACES: Translation Accuracy Challenge Sets at WMT 2023](https://mlsanity.com/paper/2311.01153/) (2023)
- [Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models](https://mlsanity.com/paper/2404.02622/) (2024)

## Abstract (arXiv)

Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis. However, recent studies suggest that these models may rely on fallible heuristics rather than deep language understanding. We introduce a challenge set to test whether NLI systems adopt one such heuristic: assuming that a sentence entails all of its subsequences, such as assuming that "Alice believes Mary is lying" entails "Alice believes Mary." We evaluate several competitive NLI models on this challenge set and find strong evidence that they do rely on the subsequence heuristic.