# A Light-weight, Effective and Efficient Model for Label Aggregation in Crowdsourcing

arXiv: https://arxiv.org/abs/2212.00007 · HTML: https://mlsanity.com/paper/2212.00007/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 8 of 8)

- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [A Survey on Truth Discovery](https://mlsanity.com/paper/1505.02463/) (2015)
- [Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing](https://mlsanity.com/paper/1411.4086/) (2014)
- [Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations](https://mlsanity.com/paper/1902.08918/) (2019)
- [Gradient Descent for Sparse Rank-One Matrix Completion for Crowd-Sourced Aggregation of Sparsely Interacting Workers](https://mlsanity.com/paper/1904.11608/) (2019)
- [A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control](https://mlsanity.com/paper/1812.02736/) (2018)
- [Streaming Bayesian Inference for Crowdsourced Classification](https://mlsanity.com/paper/1911.05712/) (2019)
- [An Unsupervised Bayesian Neural Network for Truth Discovery in Social Networks](https://mlsanity.com/paper/1906.10470/) (2019)

## Built on by (most cited first, 2 of 2)

- [Towards Long-term Annotators: A Supervised Label Aggregation Baseline](https://mlsanity.com/paper/2311.14709/) (2023)
- [A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment](https://mlsanity.com/paper/2403.08826/) (2024)

## Abstract (arXiv)

Due to the noises in crowdsourced labels, label aggregation (LA) has emerged as a standard procedure to post-process crowdsourced labels. LA methods estimate true labels from crowdsourced labels by modeling worker qualities. Most existing LA methods are iterative in nature. They need to traverse all the crowdsourced labels multiple times in order to jointly and iteratively update true labels and worker qualities until convergence. Consequently, these methods have high space and time complexities. In this paper, we treat LA as a dynamic system and model it as a Dynamic Bayesian network. From the dynamic model we derive two light-weight algorithms, LAtextsuperscriptonepass and LAtextsuperscripttwopass, which can effectively and efficiently estimate worker qualities and true labels by traversing all the labels at most twice. Due to the dynamic nature, the proposed algorithms can also estimate true labels online without re-visiting historical data. We theoretically prove the convergence property of the proposed algorithms, and bound the error of estimated worker qualities. We also analyze the space and time complexities of the proposed algorithms and show that they are equivalent to those of majority voting. Experiments conducted on 20 real-world datasets demonstrate that the proposed algorithms can effectively and efficiently aggregate labels in both offline and online settings even if they traverse all the labels at most twice.