# Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

arXiv: https://arxiv.org/abs/2302.02990 · HTML: https://mlsanity.com/paper/2302.02990/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2023.

## Built on by (most cited first, 3 of 3)

- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [A Platform for Investigating Public Health Content with Efficient Concern Classification](https://mlsanity.com/paper/2506.01308/) (2025)
- [The Shape of Power: A Multilingual Framework for Social Power Reasoning in Dialogues](https://mlsanity.com/paper/2608.28144/) (2026)

## Abstract (arXiv)

Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers' F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not - achieving mean F1 of 0.70.