mlsanityData to 7 Oct 2026

Investigating Reasons for Disagreement in Natural Language Inference

arXiv 2209.03392 ↗First posted 7 Sep 2022cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level classes. We found that some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts, leading to different interpretations of the label distribution. We explore two modeling approaches for detecting items with potential disagreement: a 4-way classification with a "Complicated" label in addition to the three standard NLI labels, and a multilabel classification approach. We found that the multilabel classification is more expressive and gives better recall of the possible interpretations in the data.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 17 of its 65 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

17 of 65 references.

References · 17

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  10. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  11. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  12. 2020Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inferencecited by 43
5 more
  1. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  2. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  3. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  4. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  5. 2021Did they answer? Subjective acts and intents in conversational discoursecited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 4; 2025: 3; 2026: 7.

29 papers.

Sort:

By area:

  1. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  2. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  3. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  4. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  5. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  6. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  7. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  8. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  9. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  10. 2023Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Designcited by 14
  11. 2023Expectations over Unspoken Alternatives Predict Pragmatic Inferencescited by 9
  12. 2023ActiveAED: A Human in the Loop Improves Annotation Error Detectioncited by 9
17 more
  1. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  2. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  3. 2023Understanding and Predicting Human Label Variation in Natural Language Inference through Explanationcited by 5
  4. 2023q2d: Turning Questions into Dialogs to Teach Models How to Searchcited by 5
  5. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inferencecited by 4
  6. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  7. 2023X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphscited by 3
  8. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
  9. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  10. 2026Humans and LLMs Diverge on Probabilistic Inferencescited by 1
  11. 2026Structured Credal Learningcited by 1
  12. 2026Quantifying and Predicting Disagreement in Graded Human Ratingscited by 1
  13. 2026Design-Based Supervised Learning with Noisy Human Labelscited by 1
  14. 2025Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understandingcited by 0
  15. 2026Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Taskscited by 0
  16. 2026Evaluating Large Language Models for Antisemitic Incident Classificationcited by 0
  17. 2026A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Codecited by 0