# Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

arXiv: https://arxiv.org/abs/2411.14103 · HTML: https://mlsanity.com/paper/2411.14103/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2024.

## Built on by (most cited first, 8 of 8)

- [A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI](https://mlsanity.com/paper/2412.13942/) (2024)
- [FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking](https://mlsanity.com/paper/2504.16188/) (2025)
- [DeepSurvey: Agent-Oriented Automated Survey Generation with Analytical Depth and Citation Reliability](https://mlsanity.com/paper/2605.29522/) (2026)
- [Inferring Adjective Hypernyms with Language Models to Increase the Connectivity of Open English Wordnet](https://mlsanity.com/paper/2506.10715/) (2025)
- [Math Natural Language Inference: this should be easy!](https://mlsanity.com/paper/2507.23063/) (2025)
- [MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference](https://mlsanity.com/paper/2510.24295/) (2025)
- [Explanation Generation for Contradiction Reconciliation with LLMs](https://mlsanity.com/paper/2603.22735/) (2026)
- [Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models](https://mlsanity.com/paper/2604.22166/) (2026)

## Abstract (arXiv)

In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.