# Human Label Variation in Implicit Discourse Relation Recognition

arXiv: https://arxiv.org/abs/2602.22723 · HTML: https://mlsanity.com/paper/2602.22723/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Did they answer? Subjective acts and intents in conversational discourse](https://mlsanity.com/paper/2104.04470/) (2021)
- [Ecologically Valid Explanations for Label Variation in NLI](https://mlsanity.com/paper/2310.13850/) (2023)
- [The Origin and Value of Disagreement Among Data Labelers: A Case Study of the Individual Difference in Hate Speech Annotation](https://mlsanity.com/paper/2112.04030/) (2021)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://mlsanity.com/paper/2304.00815/) (2023)
- [eRST: A Signaled Graph Theory of Discourse Relations and Organization](https://mlsanity.com/paper/2403.13560/) (2024)
- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [Joint Binary Neural Network for Multi-label Learning with Applications to Emotion Classification](https://mlsanity.com/paper/1802.00891/) (2018)
- [Multi-Label Classification for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2406.04461/) (2024)
- [A Survey of Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2203.02982/) (2022)
- [Towards a Progression-Aware Autonomous Dialogue Agent](https://mlsanity.com/paper/2205.03692/) (2022)

## Abstract (arXiv)

There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.