# How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLI

arXiv: https://arxiv.org/abs/2607.15870 · HTML: https://mlsanity.com/paper/2607.15870/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 48 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Builds on (most cited first, 10 of 10)

- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Can neural networks understand monotonicity reasoning?](https://mlsanity.com/paper/1906.06448/) (2019)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning](https://mlsanity.com/paper/1904.12166/) (2019)
- [Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation](https://mlsanity.com/paper/2304.12443/) (2023)
- [Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations](https://mlsanity.com/paper/2510.16458/) (2025)
- [From Disagreement to Understanding: The Case for Ambiguity Detection in NLI](https://mlsanity.com/paper/2507.15114/) (2025)
- [Optimal Sample Size Planning for the Wilcoxon-Mann-Whitney-Test](https://mlsanity.com/paper/1805.12249/) (2018)
- [Quantifying and Predicting Disagreement in Graded Human Ratings](https://mlsanity.com/paper/2605.01168/) (2026)
- [Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation](https://mlsanity.com/paper/2605.06318/) (2026)

## Built on by (most cited first, 1 of 1)

- [Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations](https://mlsanity.com/paper/2607.19231/) (2026)

## Abstract (arXiv)

Human label variation in natural language inference (NLI) is increasingly treated as a signal to be measured rather than as noise to be removed. We ask whether one candidate source of that signal, formal semantic structure such as negation, quantification, and monotonicity, changes how much annotators disagree and what they disagree about. We tag the SNLI and MNLI items of ChaosNLI, each labeled by 100 annotators, with a rule-based monotonicity tagger, check the tagger by hand on a sample of the same items, and answer three questions. At the group level, hypotheses that are not purely upward monotone attract somewhat more disagreement, but an error sensitivity analysis shows that this difference is sensitive to tagger error. At the item level, formal structure explains only a few percent of the variation and cannot pick out the items that attract high disagreement. In composition, the kinds of disagreement recorded by VariErr and LiTEx do not differ detectably across the formal boundary. Formal structure therefore belongs in the inventory of disagreement sources, with a small and bounded weight. ChaosNLI was built from low-agreement items, and every claim holds within that scope. Analysis decisions were written in a version-controlled research log before the corresponding results were computed, and negative results are reported in full.