# Investigating Reasons for Disagreement in Natural Language Inference

arXiv: https://arxiv.org/abs/2209.03392 · HTML: https://mlsanity.com/paper/2209.03392/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 65 references that are in the map.
- First posted on arXiv 7 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Evaluating Coherence in Dialogue Systems using Entailment](https://mlsanity.com/paper/1904.03371/) (2019)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [ANLIzing the Adversarial Natural Language Inference Dataset](https://mlsanity.com/paper/2010.12729/) (2020)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Learning with Different Amounts of Annotation: From Zero to Many Labels](https://mlsanity.com/paper/2109.04408/) (2021)
- [Did they answer? Subjective acts and intents in conversational discourse](https://mlsanity.com/paper/2104.04470/) (2021)

## Built on by (most cited first, 29 of 29)

- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback](https://mlsanity.com/paper/2410.19133/) (2024)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://mlsanity.com/paper/2304.00815/) (2023)
- [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://mlsanity.com/paper/2304.04758/) (2023)
- [ActiveAED: A Human in the Loop Improves Annotation Error Detection](https://mlsanity.com/paper/2305.20045/) (2023)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation](https://mlsanity.com/paper/2304.12443/) (2023)
- [q2d: Turning Questions into Dialogs to Teach Models How to Search](https://mlsanity.com/paper/2304.14318/) (2023)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Exploring Continual Learning of Compositional Generalization in NLI](https://mlsanity.com/paper/2403.04400/) (2024)
- [X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs](https://mlsanity.com/paper/2309.08873/) (2023)
- [Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale](https://mlsanity.com/paper/2504.09283/) (2025)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [Humans and LLMs Diverge on Probabilistic Inferences](https://mlsanity.com/paper/2602.23546/) (2026)
- [Structured Credal Learning](https://mlsanity.com/paper/2603.14070/) (2026)
- [Quantifying and Predicting Disagreement in Graded Human Ratings](https://mlsanity.com/paper/2605.01168/) (2026)
- [Design-Based Supervised Learning with Noisy Human Labels](https://mlsanity.com/paper/2607.15455/) (2026)
- [Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding](https://mlsanity.com/paper/2504.01132/) (2025)
- [Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks](https://mlsanity.com/paper/2605.09955/) (2026)
- [Evaluating Large Language Models for Antisemitic Incident Classification](https://mlsanity.com/paper/2607.04890/) (2026)
- [A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Code](https://mlsanity.com/paper/2609.30642/) (2026)

## Abstract (arXiv)

We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level classes. We found that some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts, leading to different interpretations of the label distribution. We explore two modeling approaches for detecting items with potential disagreement: a 4-way classification with a "Complicated" label in addition to the three standard NLI labels, and a multilabel classification approach. We found that the multilabel classification is more expressive and gives better recall of the possible interpretations in the data.