mlsanityData to 14 Oct 2026

Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

arXiv 2304.12443 ↗First posted 24 Apr 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Human label variation (Plank 2022), or annotation disagreement, exists in many natural language processing (NLP) tasks. To be robust and trusted, NLP models need to identify such variation and be able to explain it. To this end, we created the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items, each with at least 10 annotations. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

26 of 37 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  6. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  7. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  8. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  9. 2020Measuring Association Between Labels and Free-Text Rationalescited by 160
  10. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  11. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  12. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
14 more
  1. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  2. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  3. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  4. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  5. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  6. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  7. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  8. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  9. 2020e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanationscited by 31
  10. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  11. 2021On the Diversity and Limits of Human Explanationscited by 30
  12. 2022Investigating Reasons for Disagreement in Natural Language Inferencecited by 29
  13. 2020LIREx: Augmenting Language Inference with Relevant Explanationcited by 24
  14. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2024Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?cited by 15
  2. 2023From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classificationcited by 14
  3. 2025Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variationcited by 3
  4. 2025From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLPcited by 3
  5. 2026How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLIcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
  2. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference
  3. 2026Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking
  4. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
  5. 2026ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation
  6. 2022Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models
  7. 2026Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages
  8. 2025Regularization Through Reasoning: Systematic Improvements in Language Model Classification via Explanation-Enhanced Fine-Tuning