# On Crowdsourcing Task Design for Discourse Relation Annotation

arXiv: https://arxiv.org/abs/2412.11637 · HTML: https://mlsanity.com/paper/2412.11637/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 23 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 5 of 5)

- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://mlsanity.com/paper/2010.02815/) (2020)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://mlsanity.com/paper/2304.00815/) (2023)
- [Empirical Methodology for Crowdsourcing Ground Truth](https://mlsanity.com/paper/1809.08888/) (2018)
- [A Multi-Task and Multi-Label Classification Model for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2408.08971/) (2024)

## Built on by (most cited first, 2 of 2)

- [Conflicts in Texts: Data, Implications and Challenges](https://mlsanity.com/paper/2504.19472/) (2025)
- [Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2503.20588/) (2025)

## Abstract (arXiv)

Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like because or then are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource English implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus -- initially annotated with the free-choice method -- using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators' abilities to interpret and produce discourse relations.