# TaxiNLI: Taking a Ride up the NLU Hill

arXiv: https://arxiv.org/abs/2009.14505 · HTML: https://mlsanity.com/paper/2009.14505/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 27 of 27)

- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens](https://mlsanity.com/paper/2210.14011/) (2022)
- [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://mlsanity.com/paper/2110.14566/) (2021)
- [GLoRE: Evaluating Logical Reasoning of Large Language Models](https://mlsanity.com/paper/2310.09107/) (2023)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Can Transformer Language Models Predict Psychometric Properties?](https://mlsanity.com/paper/2106.06849/) (2021)
- [Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task](https://mlsanity.com/paper/2107.07229/) (2021)
- [Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks](https://mlsanity.com/paper/2204.10615/) (2022)
- [SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics](https://mlsanity.com/paper/2106.01077/) (2021)
- [Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance](https://mlsanity.com/paper/2110.02386/) (2021)
- [Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features](https://mlsanity.com/paper/2307.05454/) (2023)
- [QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning](https://mlsanity.com/paper/2205.03075/) (2022)
- [SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space](https://mlsanity.com/paper/2307.02269/) (2023)
- [LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI](https://mlsanity.com/paper/2112.02333/) (2021)
- [Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding](https://mlsanity.com/paper/2204.06283/) (2022)
- [X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs](https://mlsanity.com/paper/2309.08873/) (2023)
- [TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text](https://mlsanity.com/paper/2402.12881/) (2024)
- [Multilingual CheckList: Generation and Evaluation](https://mlsanity.com/paper/2203.12865/) (2022)
- [Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models](https://mlsanity.com/paper/2311.04659/) (2023)
- [Neural models for Factual Inconsistency Classification with Explanations](https://mlsanity.com/paper/2306.08872/) (2023)
- [Structured Prediction in NLP -- A survey](https://mlsanity.com/paper/2110.02057/) (2021)
- [Generating Intermediate Steps for NLI with Next-Step Supervision](https://mlsanity.com/paper/2208.14641/) (2022)
- [TestAug: A Framework for Augmenting Capability-based NLP Tests](https://mlsanity.com/paper/2210.08097/) (2022)
- [Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference](https://mlsanity.com/paper/2307.05034/) (2023)

## Abstract (arXiv)

Pre-trained Transformer-based neural architectures have consistently achieved state-of-the-art performance in the Natural Language Inference (NLI) task. Since NLI examples encompass a variety of linguistic, logical, and reasoning phenomena, it remains unclear as to which specific concepts are learnt by the trained systems and where they can achieve strong generalization. To investigate this question, we propose a taxonomic hierarchy of categories that are relevant for the NLI task. We introduce TAXINLI, a new dataset, that has 10k examples from the MNLI dataset (Williams et al., 2018) with these taxonomic labels. Through various experiments on TAXINLI, we observe that whereas for certain taxonomic categories SOTA neural models have achieved near perfect accuracies - a large jump over the previous models - some categories still remain difficult. Our work adds to the growing body of literature that shows the gaps in the current NLI systems and datasets through a systematic presentation and analysis of reasoning categories.