# Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

arXiv: https://arxiv.org/abs/2609.35824 · HTML: https://mlsanity.com/paper/2609.35824/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [BatchPrompt: Accomplish more with less](https://mlsanity.com/paper/2309.00384/) (2023)
- [Annotation Sensitivity: Training Data Collection Methods Affect Model Performance](https://mlsanity.com/paper/2311.14212/) (2023)
- [Position: Insights from Survey Methodology can Improve Training Data](https://mlsanity.com/paper/2403.01208/) (2024)
- [BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compression](https://mlsanity.com/paper/2503.02756/) (2025)
- [The Measurement Revolution? Credible Measurement and Inference in the Age of AI](https://mlsanity.com/paper/2608.23524/) (2026)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Scaling Technology Acceptance Analysis with Large Language Model (LLM) Annotation Systems](https://mlsanity.com/paper/2407.00702/) (2024)
- [What Is Actually Being Annotated? Inter-Prompt Reliability as a Measurement Problem in LLM-Based Social Science Labeling](https://mlsanity.com/paper/2604.16413/) (2026)
- [Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation](https://mlsanity.com/paper/2607.05554/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' kappa = 0.91). Agreement fell when we changed the task design for the same tweets (median Cohen's kappa = 0.76). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.