# Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness

arXiv: https://arxiv.org/abs/1805.04212 · HTML: https://mlsanity.com/paper/1805.04212/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2018.

## Built on by (most cited first, 17 of 17)

- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Linear Connectivity Reveals Generalization Strategies](https://mlsanity.com/paper/2205.12411/) (2022)
- [Combining Feature and Instance Attribution to Detect Artifacts](https://mlsanity.com/paper/2107.00323/) (2021)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models](https://mlsanity.com/paper/2401.14440/) (2024)
- [When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes](https://mlsanity.com/paper/2204.09722/) (2022)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [Understanding Out-of-distribution: A Perspective of Data Dynamics](https://mlsanity.com/paper/2111.14730/) (2021)
- [VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI](https://mlsanity.com/paper/2508.00965/) (2025)
- [SETI: Systematicity Evaluation of Textual Inference](https://mlsanity.com/paper/2305.15045/) (2023)
- [Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference](https://mlsanity.com/paper/2211.13331/) (2022)
- [Contrastive variational information bottleneck for aspect-based sentiment analysis](https://mlsanity.com/paper/2303.02846/) (2023)
- [Statistically Profiling Biases in Natural Language Reasoning Datasets and Models](https://mlsanity.com/paper/2102.04632/) (2021)
- [A Logic-Based Framework for Natural Language Inference in Dutch](https://mlsanity.com/paper/2110.03323/) (2021)
- [Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation](https://mlsanity.com/paper/2608.18681/) (2026)

## Abstract (arXiv)

Natural Language Inference is a challenging task that has received substantial attention, and state-of-the-art models now achieve impressive test set performance in the form of accuracy scores. Here, we go beyond this single evaluation metric to examine robustness to semantically-valid alterations to the input data. We identify three factors - insensitivity, polarity and unseen pairs - and compare their impact on three SNLI models under a variety of conditions. Our results demonstrate a number of strengths and weaknesses in the models' ability to generalise to new in-domain instances. In particular, while strong performance is possible on unseen hypernyms, unseen antonyms are more challenging for all the models. More generally, the models suffer from an insensitivity to certain small but semantically significant alterations, and are also often influenced by simple statistical correlations between words and training labels. Overall, we show that evaluations of NLI models can benefit from studying the influence of factors intrinsic to the models or found in the dataset used.