# Supporting Annotators with Affordances for Efficiently Labeling Conversational Data

arXiv: https://arxiv.org/abs/2403.07762 · HTML: https://mlsanity.com/paper/2403.07762/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2024.

## Built on by (most cited first, 3 of 3)

- [Language Models in Dialogue: Conversational Maxims for Human-AI Interactions](https://mlsanity.com/paper/2403.15115/) (2024)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review](https://mlsanity.com/paper/2605.12303/) (2026)

## Abstract (arXiv)

Without well-labeled ground truth data, machine learning-based systems would not be as ubiquitous as they are today, but these systems rely on substantial amounts of correctly labeled data. Unfortunately, crowdsourced labeling is time consuming and expensive. To address the concerns of effort and tedium, we designed CAL, a novel interface to aid in data labeling. We made several key design decisions for CAL, which include preventing inapt labels from being selected, guiding users in selecting an appropriate label when they need assistance, incorporating labeling documentation into the interface, and providing an efficient means to view previous labels. We implemented a production-quality implementation of CAL and report a user-study evaluation that compares CAL to a standard spreadsheet. Key findings of our study include users using CAL reported lower cognitive load, did not increase task time, users rated CAL to be easier to use, and users preferred CAL over the spreadsheet.