# Toward Annotator Group Bias in Crowdsourcing

arXiv: https://arxiv.org/abs/2110.08038 · HTML: https://mlsanity.com/paper/2110.08038/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [The Frontiers of Fairness in Machine Learning](https://mlsanity.com/paper/1810.08810/) (2018)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [ConStance: Modeling Annotation Contexts to Improve Stance Classification](https://mlsanity.com/paper/1708.06309/) (2017)
- [Evaluating the Crowd with Confidence](https://mlsanity.com/paper/1411.6562/) (2014)
- [Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks](https://mlsanity.com/paper/1905.04981/) (2019)

## Built on by (most cited first, 14 of 14)

- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving](https://mlsanity.com/paper/2405.01379/) (2024)
- [Explanation-based Finetuning Makes Models More Robust to Spurious Cues](https://mlsanity.com/paper/2305.04990/) (2023)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models](https://mlsanity.com/paper/2406.10130/) (2024)
- [Bias in, Bias out: Annotation Bias in Multilingual Large Language Models](https://mlsanity.com/paper/2511.14662/) (2025)
- [Augmenting Compliance-Guaranteed Customer Service Chatbots: Context-Aware Knowledge Expansion with Large Language Models](https://mlsanity.com/paper/2410.12444/) (2024)
- [Mitigating Voter Attribute Bias for Fair Opinion Aggregation](https://mlsanity.com/paper/2307.10749/) (2023)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Trustworthy Human Computation: A Survey](https://mlsanity.com/paper/2210.12324/) (2022)
- [Learnings from Technological Interventions in a Low Resource Language: Enhancing Information Access in Gondi](https://mlsanity.com/paper/2211.16172/) (2022)
- [Reducing annotator bias by belief elicitation](https://mlsanity.com/paper/2410.15726/) (2024)
- [UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers](https://mlsanity.com/paper/2608.09209/) (2026)

## Abstract (arXiv)

Crowdsourcing has emerged as a popular approach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating individual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same demographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilistic graphical framework GroupAnno to capture annotator group bias with a new extended Expectation Maximization (EM) training algorithm. We conduct experiments on both synthetic and real-world datasets. Experimental results demonstrate the effectiveness of our model in modeling annotator group bias in label aggregation and model learning over competitive baselines.