# Learning Ambiguity from Crowd Sequential Annotations

arXiv: https://arxiv.org/abs/2301.01579 · HTML: https://mlsanity.com/paper/2301.01579/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 31 references that are in the map.
- First posted on arXiv 4 Jan 2023.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [A Crowdsourced Frame Disambiguation Corpus with Ambiguity](https://mlsanity.com/paper/1904.06101/) (2019)
- [CRF-CNN: Modeling Structured Information in Human Pose Estimation](https://mlsanity.com/paper/1611.00468/) (2016)
- [Active Multi-Label Crowd Consensus](https://mlsanity.com/paper/1911.02789/) (2019)
- [Truth Discovery in Sequence Labels from Crowds](https://mlsanity.com/paper/2109.04470/) (2021)

## Built on by (most cited first, 1 of 1)

- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)

## Abstract (arXiv)

Most crowdsourcing learning methods treat disagreement between annotators as noisy labelings while inter-disagreement among experts is often a good indicator for the ambiguity and uncertainty that is inherent in natural language. In this paper, we propose a framework called Learning Ambiguity from Crowd Sequential Annotations (LA-SCA) to explore the inter-disagreement between reliable annotators and effectively preserve confusing label information. First, a hierarchical Bayesian model is developed to infer ground-truth from crowds and group the annotators with similar reliability together. By modeling the relationship between the size of group the annotator involved in, the annotator's reliability and element's unambiguity in each sequence, inter-disagreement between reliable annotators on ambiguous elements is computed to obtain label confusing information that is incorporated to cost-sensitive sequence labeling. Experimental results on POS tagging and NER tasks show that our proposed framework achieves competitive performance in inferring ground-truth from crowds and predicting unknown sequences, and interpreting hierarchical clustering results helps discover labeling patterns of annotators with similar reliability.