# Data Quality in Crowdsourcing and Spamming Behavior Detection

arXiv: https://arxiv.org/abs/2404.17582 · HTML: https://mlsanity.com/paper/2404.17582/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 66 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 2 of 2)

- [ColluEagle: Collusive review spammer detection using Markov random fields](https://mlsanity.com/paper/1911.01690/) (2019)
- [Evaluating Trustworthiness of AI-Enabled Decision Support Systems: Validation of the Multisource AI Scorecard Table (MAST)](https://mlsanity.com/paper/2311.18040/) (2023)

## Built on by (most cited first, 2 of 2)

- [Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions](https://mlsanity.com/paper/2509.08217/) (2025)
- [Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge](https://mlsanity.com/paper/2610.14029/) (2026)

## Abstract (arXiv)

As crowdsourcing emerges as an efficient and cost-effective method for obtaining labels for machine learning datasets, it is important to assess the quality of crowd-provided data, so as to improve analysis performance and reduce biases in subsequent machine learning tasks. Given the lack of ground truth in most cases of crowdsourcing, we refer to data quality as annotators' consistency and credibility. Unlike the simple scenarios where Kappa coefficient and intraclass correlation coefficient usually can apply, online crowdsourcing requires dealing with more complex situations. We introduce a systematic method for evaluating data quality and detecting spamming threats via variance decomposition, and we classify spammers into three categories based on their different behavioral patterns. A spammer index is proposed to assess entire data consistency, and two metrics are developed to measure crowd workers' credibility by utilizing the Markov chain and generalized random effects models. Furthermore, we showcase the practicality of our techniques and their advantages by applying them on a face verification task with both simulation and real-world data collected from two crowdsourcing platforms.