# Incorporating Annotator Uncertainty into Representations of Discourse Relations

arXiv: https://arxiv.org/abs/2308.07179 · HTML: https://mlsanity.com/paper/2308.07179/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 31 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 3 of 3)

- [Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure](https://mlsanity.com/paper/2004.05080/) (2020)
- [QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://mlsanity.com/paper/2010.02815/) (2020)
- [The distribution of discourse relations within and across turns in spontaneous conversation](https://mlsanity.com/paper/2307.03645/) (2023)

## Abstract (arXiv)

Annotation of discourse relations is a known difficult task, especially for non-expert annotators. In this paper, we investigate novice annotators' uncertainty on the annotation of discourse relations on spoken conversational data. We find that dialogue context (single turn, pair of turns within speaker, and pair of turns across speakers) is a significant predictor of confidence scores. We compute distributed representations of discourse relations from co-occurrence statistics that incorporate information about confidence scores and dialogue context. We perform a hierarchical clustering analysis using these representations and show that weighting discourse relation representations with information about confidence and dialogue context coherently models our annotators' uncertainty about discourse relation labels.