# ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation

arXiv: https://arxiv.org/abs/2603.21094 · HTML: https://mlsanity.com/paper/2603.21094/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 3 of its 15 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 3 of 3)

- [Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI](https://mlsanity.com/paper/2409.09467/) (2024)
- [Bias in the Loop: How Humans Evaluate AI-Generated Suggestions](https://mlsanity.com/paper/2509.08514/) (2025)
- [Collaborative Human-AI Risk Annotation: Co-Annotating Online Incivility with CHAIRA](https://mlsanity.com/paper/2409.14223/) (2024)

## Abstract (arXiv)

Human annotation is central to NLP evaluation, yet subjective tasks often exhibit substantial variability across annotators. While large language models (LLMs) can provide structured reasoning to support annotation, their influence on human annotation behavior remains underexplored. We introduce ReasonScaffold, a scaffolded reasoning annotation protocol that exposes LLM-generated explanations while withholding predicted labels. We study how reasoning affects human annotation behavior in a controlled setting, rather than evaluating annotation accuracy. Using a two-pass protocol inspired by Delphi-style revision, annotators first label instances independently and then revise their decisions after viewing model-generated reasoning. We evaluate the approach on sentiment classification and opinion detection tasks, analyzing changes in inter-annotator agreement and revision behavior. To quantify these effects, we introduce the Annotator Effort Proxy (AEP), a metric capturing the proportion of labels revised after exposure to reasoning. Our results show that exposure to reasoning is associated with increased agreement, along with minimal revision, suggesting that reasoning helps resolve ambiguous cases without inducing widespread changes. These findings provide insight into how reasoning explanations shape annotation consistency and highlight reasoning-based scaffolds as a practical mechanism for human--AI co-annotation workflows.