# Humans and language models diverge when predicting repeating text

arXiv: https://arxiv.org/abs/2310.06408 · HTML: https://mlsanity.com/paper/2310.06408/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Criticality in Formal Languages and Statistical Physics](https://mlsanity.com/paper/1606.06737/) (2016)
- [One Model for the Learning of Language](https://mlsanity.com/paper/1711.06301/) (2017)
- [Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling](https://mlsanity.com/paper/2009.03954/) (2020)
- [Multi-timescale Representation Learning in LSTM Language Models](https://mlsanity.com/paper/2009.12727/) (2020)
- [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://mlsanity.com/paper/2210.13569/) (2022)

## Built on by (most cited first, 13 of 13)

- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned](https://mlsanity.com/paper/2410.12057/) (2024)
- [Transformer verbatim in-context retrieval across time and scale](https://mlsanity.com/paper/2411.07075/) (2024)
- [Interpretable Next-token Prediction via the Generalized Induction Head](https://mlsanity.com/paper/2411.00066/) (2024)
- [Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models](https://mlsanity.com/paper/2508.05803/) (2025)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)
- [Simulating Human Memory with Language Models](https://mlsanity.com/paper/2605.25680/) (2026)
- [Can LLMs capture stable human-generated sentence entropy measures?](https://mlsanity.com/paper/2602.04570/) (2026)
- [On the scaling relationship between cloze probabilities and language model next-token prediction](https://mlsanity.com/paper/2602.17848/) (2026)
- [Automatic or Controlled? Repetition Priming Reveals Divergent Processing in Base LLMs, Instruct LLMs, and Humans](https://mlsanity.com/paper/2608.14681/) (2026)
- [Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual Influence](https://mlsanity.com/paper/2610.04791/) (2026)

## Abstract (arXiv)

Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges. We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role. We traced the cause of this divergence to specific attention heads in a middle layer. Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans. We hope that this scenario will spur future work in bringing LMs closer to human behavior.