# Testing the Generalization Power of Neural Network Models Across NLI Benchmarks

arXiv: https://arxiv.org/abs/1810.09774 · HTML: https://mlsanity.com/paper/1810.09774/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2018.

## Built on by (most cited first, 15 of 15)

- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [FacTeR-Check: Semi-automated fact-checking through Semantic Similarity and Natural Language Inference](https://mlsanity.com/paper/2110.14532/) (2021)
- [ER-Test: Evaluating Explanation Regularization Methods for Language Models](https://mlsanity.com/paper/2205.12542/) (2022)
- [NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance](https://mlsanity.com/paper/2104.04751/) (2021)
- [DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines](https://mlsanity.com/paper/2212.10557/) (2022)
- [Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging](https://mlsanity.com/paper/2304.04726/) (2023)
- [OYXOY: A Modern NLP Test Suite for Modern Greek](https://mlsanity.com/paper/2309.07009/) (2023)
- [Early Period of Training Impacts Adaptation for Out-of-Distribution Generalization: An Empirical Study](https://mlsanity.com/paper/2403.15210/) (2024)
- [How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets](https://mlsanity.com/paper/2201.04467/) (2022)
- [Grounded learning for compositional vector semantics](https://mlsanity.com/paper/2401.06808/) (2024)
- [Dataset Creation for Visual Entailment using Generative AI](https://mlsanity.com/paper/2508.11605/) (2025)
- [ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery](https://mlsanity.com/paper/2605.16902/) (2026)
- [LLM and Human Modes of Representation](https://mlsanity.com/paper/2606.21616/) (2026)

## Abstract (arXiv)

Neural network models have been very successful in natural language inference, with the best models reaching 90% accuracy in some benchmarks. However, the success of these models turns out to be largely benchmark specific. We show that models trained on a natural language inference dataset drawn from one benchmark fail to perform well in others, even if the notion of inference assumed in these benchmarks is the same or similar. We train six high performing neural network models on different datasets and show that each one of these has problems of generalizing when we replace the original test set with a test set taken from another corpus designed for the same task. In light of these results, we argue that most of the current neural network models are not able to generalize well in the task of natural language inference. We find that using large pre-trained language models helps with transfer learning when the datasets are similar enough. Our results also highlight that the current NLI datasets do not cover the different nuances of inference extensively enough.