# Ecologically Valid Explanations for Label Variation in NLI

arXiv: https://arxiv.org/abs/2310.13850 · HTML: https://mlsanity.com/paper/2310.13850/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations](https://mlsanity.com/paper/1910.03065/) (2019)
- [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [FLamE: Few-shot Learning from Natural Language Explanations](https://mlsanity.com/paper/2306.08042/) (2023)

## Built on by (most cited first, 16 of 16)

- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- ["Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?](https://mlsanity.com/paper/2406.17600/) (2024)
- [A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI](https://mlsanity.com/paper/2412.13942/) (2024)
- [Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations](https://mlsanity.com/paper/2510.16458/) (2025)
- [LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference](https://mlsanity.com/paper/2505.22848/) (2025)
- [Revisiting Active Learning under (Human) Label Variation](https://mlsanity.com/paper/2507.02593/) (2025)
- [From Disagreement to Understanding: The Case for Ambiguity Detection in NLI](https://mlsanity.com/paper/2507.15114/) (2025)
- [From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP](https://mlsanity.com/paper/2601.07988/) (2026)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Humans and LLMs Diverge on Probabilistic Inferences](https://mlsanity.com/paper/2602.23546/) (2026)
- [Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization](https://mlsanity.com/paper/2605.28802/) (2026)
- [Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection](https://mlsanity.com/paper/2605.31563/) (2026)
- [DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning](https://mlsanity.com/paper/2509.09524/) (2025)
- [EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI](https://mlsanity.com/paper/2511.08949/) (2025)
- [Human Label Variation in Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2602.22723/) (2026)
- [Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations](https://mlsanity.com/paper/2606.25383/) (2026)

## Abstract (arXiv)

Human label variation, or annotation disagreement, exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items (at least 10 explanations per item). The LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation: annotators sometimes choose the same label for different reasons. This suggests that explanations are crucial for navigating label interpretations in general. We few-shot prompt large language models to generate explanations but the results are inconsistent: they sometimes produces valid and informative explanations, but it also generates implausible ones that do not support the label, highlighting directions for improvement.