# GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling

arXiv: https://arxiv.org/abs/2003.00475 · HTML: https://mlsanity.com/paper/2003.00475/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2020.

## Built on by (most cited first, 3 of 3)

- [Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems](https://mlsanity.com/paper/2307.12975/) (2023)
- [Subjective and Objective Quality Assessment of Mobile Gaming Video](https://mlsanity.com/paper/2103.05099/) (2021)
- [Granular feedback merits sophisticated aggregation](https://mlsanity.com/paper/2507.12041/) (2025)

## Abstract (arXiv)

In the big data era, data labeling can be obtained through crowdsourcing. Nevertheless, the obtained labels are generally noisy, unreliable or even adversarial. In this paper, we propose a probabilistic graphical annotation model to infer the underlying ground truth and annotator's behavior. To accommodate both discrete and continuous application scenarios (e.g., classifying scenes vs. rating videos on a Likert scale), the underlying ground truth is considered following a distribution rather than a single value. In this way, the reliable but potentially divergent opinions from "good" annotators can be recovered. The proposed model is able to identify whether an annotator has worked diligently towards the task during the labeling procedure, which could be used for further selection of qualified annotators. Our model has been tested on both simulated data and real-world data, where it always shows superior performance than the other state-of-the-art models in terms of accuracy and robustness.