# Cross-domain-aware Worker Selection with Training for Crowdsourced Annotation

arXiv: https://arxiv.org/abs/2406.06977 · HTML: https://mlsanity.com/paper/2406.06977/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 56 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Dynamic Key-Value Memory Networks for Knowledge Tracing](https://mlsanity.com/paper/1611.08108/) (2016)
- [Knowledge Tracing: A Survey](https://mlsanity.com/paper/2201.06953/) (2022)
- [Knowledge Tracing with Sequential Key-Value Memory Networks](https://mlsanity.com/paper/1910.13197/) (2019)
- [Deep Knowledge Tracing and Dynamic Student Classification for Knowledge Tracing](https://mlsanity.com/paper/1809.08713/) (2018)
- [Back to the Basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation](https://mlsanity.com/paper/1604.02336/) (2016)
- [Cheaper and Better: Selecting Good Workers for Crowdsourcing](https://mlsanity.com/paper/1502.00725/) (2015)

## Built on by (most cited first, 2 of 2)

- [KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering](https://mlsanity.com/paper/2509.04716/) (2025)
- [AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling](https://mlsanity.com/paper/2607.18465/) (2026)

## Abstract (arXiv)

Annotation through crowdsourcing draws incremental attention, which relies on an effective selection scheme given a pool of workers. Existing methods propose to select workers based on their performance on tasks with ground truth, while two important points are missed. 1) The historical performances of workers in other tasks. In real-world scenarios, workers need to solve a new task whose correlation with previous tasks is not well-known before the training, which is called cross-domain. 2) The dynamic worker performance as workers will learn from the ground truth. In this paper, we consider both factors in designing an allocation scheme named cross-domain-aware worker selection with training approach. Our approach proposes two estimation modules to both statistically analyze the cross-domain correlation and simulate the learning gain of workers dynamically. A framework with a theoretical analysis of the worker elimination process is given. To validate the effectiveness of our methods, we collect two novel real-world datasets and generate synthetic datasets. The experiment results show that our method outperforms the baselines on both real-world and synthetic datasets.