# Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

arXiv: https://arxiv.org/abs/2304.00815 · HTML: https://mlsanity.com/paper/2304.00815/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 72 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis](https://mlsanity.com/paper/2205.01996/) (2022)
- [Detecting Stance in Media on Global Warming](https://mlsanity.com/paper/2010.15149/) (2020)
- [Large-Scale QA-SRL Parsing](https://mlsanity.com/paper/1805.05377/) (2018)
- [Uncertain Natural Language Inference](https://mlsanity.com/paper/1909.03042/) (2019)
- [QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://mlsanity.com/paper/2010.02815/) (2020)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://mlsanity.com/paper/2209.03392/) (2022)
- [Learning with Different Amounts of Annotation: From Zero to Many Labels](https://mlsanity.com/paper/2109.04408/) (2021)
- [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018)
- [Did they answer? Subjective acts and intents in conversational discourse](https://mlsanity.com/paper/2104.04470/) (2021)
- [Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections](https://mlsanity.com/paper/2111.00701/) (2021)
- [Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1811.01697/) (2018)
- [Bridging Anaphora Resolution as Question Answering](https://mlsanity.com/paper/2004.07898/) (2020)
- [Text-based NP Enrichment](https://mlsanity.com/paper/2109.12085/) (2021)
- [Empirical Methodology for Crowdsourcing Ground Truth](https://mlsanity.com/paper/1809.08888/) (2018)
- [Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection](https://mlsanity.com/paper/1704.05753/) (2017)

## Built on by (most cited first, 14 of 14)

- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations](https://mlsanity.com/paper/2402.01423/) (2024)
- [PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm](https://mlsanity.com/paper/2601.08951/) (2026)
- [Promptly Predicting Structures: The Return of Inference](https://mlsanity.com/paper/2401.06877/) (2024)
- [Explicating the Implicit: Argument Detection Beyond Sentence Boundaries](https://mlsanity.com/paper/2408.04246/) (2024)
- [A Multi-Task and Multi-Label Classification Model for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2408.08971/) (2024)
- [Discursive Circuits: How Do Language Models Understand Discourse Relations?](https://mlsanity.com/paper/2510.11210/) (2025)
- [What's Hard in English RST Parsing? Predictive Models for Error Analysis](https://mlsanity.com/paper/2309.04940/) (2023)
- [AI for All: Identifying AI incidents Related to Diversity and Inclusion](https://mlsanity.com/paper/2408.01438/) (2024)
- [On Crowdsourcing Task Design for Discourse Relation Annotation](https://mlsanity.com/paper/2412.11637/) (2024)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2503.20588/) (2025)
- [Multi-Lingual Implicit Discourse Relation Recognition with Multi-Label Hierarchical Learning](https://mlsanity.com/paper/2508.20712/) (2025)
- [Human Label Variation in Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2602.22723/) (2026)

## Abstract (arXiv)

Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias: task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations' ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.