# Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes

arXiv: https://arxiv.org/abs/2511.21912 · HTML: https://mlsanity.com/paper/2511.21912/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention](https://mlsanity.com/paper/2010.07891/) (2020)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Geometric-Averaged Preference Optimization for Soft Preference Labels](https://mlsanity.com/paper/2409.06691/) (2024)
- [Data-Centric Human Preference with Rationales for Direct Preference Alignment](https://mlsanity.com/paper/2407.14477/) (2024)
- [RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment](https://mlsanity.com/paper/2509.24159/) (2025)

## Abstract (arXiv)

We propose an annotation approach that captures not only labels but also the reading process underlying annotators' decisions, e.g., what parts of the text they focus on, re-read or skim. Using this framework, we conduct a case study on the preference annotation task, creating a dataset PreferRead that contains fine-grained annotator reading behaviors obtained from mouse tracking. PreferRead enables detailed analysis of how annotators navigate between a prompt and two candidate responses before selecting their preference. We find that annotators re-read a response in roughly half of all trials, most often revisiting the option they ultimately choose, and rarely revisit the prompt. Reading behaviors are also significantly related to annotation outcomes: re-reading is associated with higher inter-annotator agreement, whereas long reading paths and times are associated with lower agreement. These results demonstrate that reading processes provide a complementary cognitive dimension for understanding annotator reliability, decision-making and disagreement in complex, subjective NLP tasks. Our code and data are publicly available.