# Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations

arXiv: https://arxiv.org/abs/1902.08918 · HTML: https://mlsanity.com/paper/1902.08918/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2019.

## Built on by (most cited first, 11 of 11)

- [Trustworthy Knowledge Graph Completion Based on Multi-sourced Noisy Data](https://mlsanity.com/paper/2201.08580/) (2022)
- [A Light-weight, Effective and Efficient Model for Label Aggregation in Crowdsourcing](https://mlsanity.com/paper/2212.00007/) (2022)
- [Mitigating Voter Attribute Bias for Fair Opinion Aggregation](https://mlsanity.com/paper/2307.10749/) (2023)
- [Mitigating Observation Biases in Crowdsourced Label Aggregation](https://mlsanity.com/paper/2302.13100/) (2023)
- [Temporal-aware Language Representation Learning From Crowdsourced Labels](https://mlsanity.com/paper/2107.07958/) (2021)
- [Towards AI-Empowered Crowdsourcing](https://mlsanity.com/paper/2212.14676/) (2022)
- [A Phone-based Distributed Ambient Temperature Measurement System with An Efficient Label-free Automated Training Strategy](https://mlsanity.com/paper/2404.10401/) (2024)
- [The Majority Vote Paradigm Shift: When Popular Meets Optimal](https://mlsanity.com/paper/2502.12581/) (2025)
- [Understanding the Essence: Delving into Annotator Prototype Learning for Multi-Class Annotation Aggregation](https://mlsanity.com/paper/2508.02123/) (2025)
- [AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling](https://mlsanity.com/paper/2607.18465/) (2026)
- [Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator](https://mlsanity.com/paper/2608.22432/) (2026)

## Abstract (arXiv)

Crowd-sourcing is a cheap and popular means of creating training and evaluation datasets for machine learning, however it poses the problem of `truth inference', as individual workers cannot be wholly trusted to provide reliable annotations. Research into models of annotation aggregation attempts to infer a latent `true' annotation, which has been shown to improve the utility of crowd-sourced data. However, existing techniques beat simple baselines only in low redundancy settings, where the number of annotations per instance is low (≤ 3), or in situations where workers are unreliable and produce low quality annotations (e.g., through spamming, random, or adversarial behaviours.) As we show, datasets produced by crowd-sourcing are often not of this type: the data is highly redundantly annotated (≥ 5 annotations per instance), and the vast majority of workers produce high quality outputs. In these settings, the majority vote heuristic performs very well, and most truth inference models underperform this simple baseline. We propose a novel technique, based on a Bayesian graphical model with conjugate priors, and simple iterative expectation-maximisation inference. Our technique produces competitive performance to the state-of-the-art benchmark methods, and is the only method that significantly outperforms the majority vote heuristic at one-sided level 0.025, shown by significance tests. Moreover, our technique is simple, is implemented in only 50 lines of code, and trains in seconds.