# Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study

arXiv: https://arxiv.org/abs/2510.12835 · HTML: https://mlsanity.com/paper/2510.12835/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 12 of 12)

- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [BioRED: A Rich Biomedical Relation Extraction Dataset](https://mlsanity.com/paper/2204.04263/) (2022)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [TeamTat: a collaborative text annotation tool](https://mlsanity.com/paper/2004.11894/) (2020)

## Built on by (most cited first, 2 of 2)

- [Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs](https://mlsanity.com/paper/2606.28574/) (2026)
- [Refining and Reusing Annotation Guidelines for LLM Annotation](https://mlsanity.com/paper/2605.20809/) (2026)

## Abstract (arXiv)

This study investigates how existing annotation guidelines can be repurposed to instruct large language model (LLM) annotators for text annotation tasks. Traditional guidelines are written for human annotators who internalize training, while LLMs require explicit, structured instructions. We propose a moderation-oriented guideline repurposing method that transforms guidelines into clear directives for LLMs through an LLM moderation process. Using the NCBI Disease Corpus as a case study, our experiments show that repurposed guidelines can effectively guide LLM annotators, while revealing several practical challenges. The results highlight the potential of this workflow to support scalable and cost-effective refinement of annotation guidelines and automated annotation.