# ANLIzing the Adversarial Natural Language Inference Dataset

arXiv: https://arxiv.org/abs/2010.12729 · HTML: https://mlsanity.com/paper/2010.12729/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 30 of 36)

- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [SafeText: A Benchmark for Exploring Physical Safety in Language Models](https://mlsanity.com/paper/2210.10045/) (2022)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [Does Putting a Linguist in the Loop Improve NLU Data Collection?](https://mlsanity.com/paper/2104.07179/) (2021)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://mlsanity.com/paper/2209.03392/) (2022)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)
- [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://mlsanity.com/paper/2110.14566/) (2021)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Long-form evaluation of model editing](https://mlsanity.com/paper/2402.09394/) (2024)
- [ALERT: Adapting Language Models to Reasoning Tasks](https://mlsanity.com/paper/2212.08286/) (2022)
- [Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks](https://mlsanity.com/paper/2204.01906/) (2022)
- [ARGUS: Hallucination and Omission Evaluation in Video-LLMs](https://mlsanity.com/paper/2506.07371/) (2025)
- [He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics](https://mlsanity.com/paper/2107.00807/) (2021)
- [Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair](https://mlsanity.com/paper/2111.08181/) (2021)
- [What Makes Reading Comprehension Questions Difficult?](https://mlsanity.com/paper/2203.06342/) (2022)
- [Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers](https://mlsanity.com/paper/2211.11586/) (2022)
- [JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset](https://mlsanity.com/paper/2212.03419/) (2022)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks](https://mlsanity.com/paper/2204.10615/) (2022)
- [longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks](https://mlsanity.com/paper/2206.14729/) (2022)
- [Training Large Language Models Efficiently with Sparsity and Dataflow](https://mlsanity.com/paper/2304.05511/) (2023)
- [LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents](https://mlsanity.com/paper/2510.03418/) (2025)
- [Knowledge-Enhanced Evidence Retrieval for Counterargument Generation](https://mlsanity.com/paper/2109.09057/) (2021)
- [MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types](https://mlsanity.com/paper/2306.10452/) (2023)
- [Analyzing Multi-Task Learning for Abstractive Text Summarization](https://mlsanity.com/paper/2210.14606/) (2022)
- [Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference](https://mlsanity.com/paper/2211.03862/) (2022)
- [Who's the Best Detective? LLMs vs. MLs in Detecting Incoherent Fourth Grade Math Answers](https://mlsanity.com/paper/2304.11257/) (2023)
- [Giving AI Personalities Leads to More Human-Like Reasoning](https://mlsanity.com/paper/2502.14155/) (2025)

## Abstract (arXiv)

We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over multiple rounds. We propose a fine-grained annotation scheme of the different aspects of inference that are responsible for the gold classification labels, and use it to hand-code all three of the ANLI development sets. We use these annotations to answer a variety of interesting questions: which inference types are most common, which models have the highest performance on each reasoning type, and which types are the most challenging for state of-the-art models? We hope that our annotations will enable more fine-grained evaluation of models trained on ANLI, provide us with a deeper understanding of where models fail and succeed, and help us determine how to train better models in future.