# CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

arXiv: https://arxiv.org/abs/1808.06080 · HTML: https://mlsanity.com/paper/1808.06080/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2018.

## Built on by (most cited first, 27 of 27)

- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive](https://mlsanity.com/paper/2301.12534/) (2023)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://mlsanity.com/paper/2304.00815/) (2023)
- [Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science](https://mlsanity.com/paper/2203.07031/) (2022)
- [Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?](https://mlsanity.com/paper/2408.14141/) (2024)
- [Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions](https://mlsanity.com/paper/2101.11956/) (2021)
- [Rater Cohesion and Quality from a Vicarious Perspective](https://mlsanity.com/paper/2408.08411/) (2024)
- [NUTMEG: Separating Signal From Noise in Annotator Disagreement](https://mlsanity.com/paper/2507.18890/) (2025)
- [Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities](https://mlsanity.com/paper/2311.00203/) (2023)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers](https://mlsanity.com/paper/2112.02255/) (2021)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline](https://mlsanity.com/paper/2111.03511/) (2021)
- [Finding the Ground-Truth from Multiple Labellers: Why Parameters of the Task Matter](https://mlsanity.com/paper/2102.08482/) (2021)
- [Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media](https://mlsanity.com/paper/2410.14515/) (2024)
- [Subjective Logic Encodings](https://mlsanity.com/paper/2502.12225/) (2025)
- [Efficient Annotator Reliability Assessment with EffiARA](https://mlsanity.com/paper/2504.00589/) (2025)
- [Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models](https://mlsanity.com/paper/2608.17183/) (2026)
- [Analyzing social media with crowdsourcing in Crowd4SDG](https://mlsanity.com/paper/2208.02689/) (2022)
- [SCRum-9: Multilingual Stance Classification over Rumours on Social Media](https://mlsanity.com/paper/2505.18916/) (2025)
- [Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions](https://mlsanity.com/paper/2509.08217/) (2025)
- [Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework](https://mlsanity.com/paper/2510.04366/) (2025)
- [Improving ML Training Data with Gold-Standard Quality Metrics](https://mlsanity.com/paper/2512.20577/) (2025)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering](https://mlsanity.com/paper/2604.01344/) (2026)
- [ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation](https://mlsanity.com/paper/2606.21867/) (2026)
- [Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment](https://mlsanity.com/paper/2608.30842/) (2026)

## Abstract (arXiv)

Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system -- worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core .