mlsanityData to 7 Oct 2026

Humans and language models diverge when predicting repeating text

arXiv 2310.06408 ↗First posted 10 Oct 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges. We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role. We traced the cause of this divergence to specific attention heads in a middle layer. Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans. We hope that this scenario will spur future work in bringing LMs closer to human behavior.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

16 of 36 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  9. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  10. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  11. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  12. 2016Criticality in Formal Languages and Statistical Physicscited by 33
4 more
  1. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  2. 2017One Model for the Learning of Languagecited by 21
  3. 2020Multi-timescale Representation Learning in LSTM Language Modelscited by 13
  4. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 5.

12 papers.

  1. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  2. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  3. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  4. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 4
  5. 2024Transformer verbatim in-context retrieval across time and scalecited by 3
  6. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3
  7. 2024Interpretable Next-token Prediction via the Generalized Induction Headcited by 2
  8. 2026Simulating Human Memory with Language Modelscited by 2
  9. 2026Can LLMs capture stable human-generated sentence entropy measures?cited by 1
  10. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  11. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  12. 2026Automatic or Controlled? Repetition Priming Reveals Divergent Processing in Base LLMs, Instruct LLMs, and Humanscited by 0