# In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers

arXiv: https://arxiv.org/abs/2112.02255 · HTML: https://mlsanity.com/paper/2112.02255/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 83 references that are in the map.
- First posted on arXiv 4 Dec 2021.

## Builds on (most cited first, 7 of 7)

- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [Crowdsourcing in Computer Vision](https://mlsanity.com/paper/1611.02145/) (2016)
- [Discovering and Validating AI Errors With Crowdsourced Failure Reports](https://mlsanity.com/paper/2109.11690/) (2021)
- [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018)
- [Diversity in Sociotechnical Machine Learning Systems](https://mlsanity.com/paper/2107.09163/) (2021)
- [Mathematical Foundations for Social Computing](https://mlsanity.com/paper/2007.03661/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Large Language Model Approach to Educational Survey Feedback Analysis](https://mlsanity.com/paper/2309.17447/) (2023)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [A Culturally-Aware Tool for Crowdworkers: Leveraging Chronemics to Support Diverse Work Styles](https://mlsanity.com/paper/2408.07838/) (2024)
- [Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework](https://mlsanity.com/paper/2510.04366/) (2025)
- [iPOE: Interpretable Prompt Optimization via Explanations](https://mlsanity.com/paper/2605.18113/) (2026)

## Abstract (arXiv)

We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality. Stage 1 (FIND) asks the crowd to find examples whose correct label seems ambiguous given task instructions. Workers are also asked to provide a short tag which describes the ambiguous concept embodied by the specific instance found. We compare collaborative vs. non-collaborative designs for this stage. In Stage 2 (RESOLVE), the requester selects one or more of these ambiguous examples to label (resolving ambiguity). The new label(s) are automatically injected back into task instructions in order to improve clarity. Finally, in Stage 3 (LABEL), workers perform the actual annotation using the revised guidelines with clarifying examples. We compare three designs for using these examples: examples only, tags only, or both. We report image labeling experiments over six task designs using Amazon's Mechanical Turk. Results show improved annotation accuracy and further insights regarding effective design for crowdsourced annotation tasks.