# Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models

arXiv: https://arxiv.org/abs/2005.14709 · HTML: https://mlsanity.com/paper/2005.14709/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 11 of 11)

- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks](https://mlsanity.com/paper/2210.10040/) (2022)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [Towards Human-Centred Explainability Benchmarks For Text Classification](https://mlsanity.com/paper/2211.05452/) (2022)
- [Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach](https://mlsanity.com/paper/2312.09718/) (2023)
- [Learning to Generate and Evaluate Fact-checking Explanations with Transformers](https://mlsanity.com/paper/2410.15669/) (2024)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [False perfection in machine prediction: Detecting and assessing circularity problems in machine learning](https://mlsanity.com/paper/2106.12417/) (2021)

## Abstract (arXiv)

Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope it will be a useful resource for researchers who propose new datasets, to have a set of tools to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who develop novel architectures, to further understand the implications of their improvements with respect to their model's acquired capabilities.