mlsanityData to 7 Oct 2026

CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

arXiv 1808.06080 ↗First posted 18 Aug 2018cs.HC cs.SIcited by 27

Not on the map. Why?

Abstract (arXiv)

Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system -- worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core .

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 5; 2024: 4; 2025: 7; 2026: 5.

27 papers.

Sort:

By area:

  1. 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 28
  2. 2023Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensivecited by 16
  3. 2023Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Designcited by 14
  4. 2022Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Sciencecited by 13
  5. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  6. 2021Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotionscited by 9
  7. 2024Rater Cohesion and Quality from a Vicarious Perspectivecited by 8
  8. 2023Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communitiescited by 6
  9. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  10. 2021In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workerscited by 5
  11. 2023Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collectioncited by 5
  12. 2025NUTMEG: Separating Signal From Noise in Annotator Disagreementcited by 5
15 more
  1. 2021Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipelinecited by 3
  2. 2021Finding the Ground-Truth from Multiple Labellers: Why Parameters of the Task Mattercited by 1
  3. 2024Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Mediacited by 1
  4. 2025Subjective Logic Encodingscited by 1
  5. 2025Efficient Annotator Reliability Assessment with EffiARAcited by 1
  6. 2026Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Modelscited by 1
  7. 2022Analyzing social media with crowdsourcing in Crowd4SDGcited by 0
  8. 2025SCRum-9: Multilingual Stance Classification over Rumours on Social Mediacited by 0
  9. 2025Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributionscited by 0
  10. 2025Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Frameworkcited by 0
  11. 2025Improving ML Training Data with Gold-Standard Quality Metricscited by 0
  12. 2026Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLPcited by 0
  13. 2026IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineeringcited by 0
  14. 2026ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotationcited by 0
  15. 2026Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignmentcited by 0