# Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

arXiv: https://arxiv.org/abs/2305.06683 · HTML: https://mlsanity.com/paper/2305.06683/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 5 of 5)

- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?](https://mlsanity.com/paper/2106.00794/) (2021)
- [Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations](https://mlsanity.com/paper/2204.10714/) (2022)
- [Top K Ranking for Multi-Armed Bandit with Noisy Evaluations](https://mlsanity.com/paper/2112.06517/) (2021)
- [Harnessing Context for Budget-Limited Crowdsensing with Massive Uncertain Workers](https://mlsanity.com/paper/2107.01385/) (2021)

## Abstract (arXiv)

This paper introduces a novel crowdsourcing worker selection algorithm, enhancing annotation quality and reducing costs. Unlike previous studies targeting simpler tasks, this study contends with the complexities of label interdependencies in sequence labeling. The proposed algorithm utilizes a Combinatorial Multi-Armed Bandit (CMAB) approach for worker selection, and a cost-effective human feedback mechanism. The challenge of dealing with imbalanced and small-scale datasets, which hinders offline simulation of worker selection, is tackled using an innovative data augmentation method termed shifting, expanding, and shrinking (SES). Rigorous testing on CoNLL 2003 NER and Chinese OEI datasets showcased the algorithm's efficiency, with an increase in F1 score up to 100.04% of the expert-only baseline, alongside cost savings up to 65.97%. The paper also encompasses a dataset-independent test emulating annotation evaluation through a Bernoulli distribution, which still led to an impressive 97.56% F1 score of the expert baseline and 59.88% cost savings. Furthermore, our approach can be seamlessly integrated into Reinforcement Learning from Human Feedback (RLHF) systems, offering a cost-effective solution for obtaining human feedback.