mlsanityData to 14 Oct 2026

To model human linguistic prediction, make LLMs less superhuman

arXiv 2510.05141 ↗First posted 1 Oct 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

When we read, we make predictions about upcoming words; these predictions influence our reading behavior. The success of large language models (LLMs), which, like humans, make predictions about upcoming words, has motivated their use as models of human linguistic prediction. Surprisingly, in the last few years, as LLMs' ability to predict the next word has improved, their ability to explain reading behavior has declined. We argue this is because current LLMs can predict upcoming words much better than human readers can. This 'superhumanness' is driven by LLMs' extensive training data, stronger long-term memory of training examples, and stronger short-term memory. We advocate for LLMs with human-like memory and for new experiments to measure the alignment between humans and LLMs, and outline directions towards achieving these goals.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 40 of its 102 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

40 of 102 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  11. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  12. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
28 more
  1. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  2. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  3. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  4. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  5. 2016Exploring the Limits of Language Modelingcited by 173
  6. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  7. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  8. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  9. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  10. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  11. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  12. 2017The Natural Stories Corpuscited by 40
  13. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  14. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 35
  15. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  16. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  17. 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 27
  18. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25
  19. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  20. 2024Causal Estimation of Memorisation Profilescited by 20
  21. 2023Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortexcited by 18
  22. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  23. 2023Humans and language models diverge when predicting repeating textcited by 13
  24. 2025Better artificial intelligence does not mean better models of biologycited by 10
  25. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8
  26. 2024The Effect of Surprisal on Reading Times in Information Seeking and Repeated Readingcited by 8
  27. 2023Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?cited by 7
  28. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  2. 2026Simulating Human Memory with Language Modelscited by 2
  3. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  4. 2026LLM surprisal is necessary but not sufficient to capture English garden-path effects: Evidence from joint latent modeling of reading paradigmscited by 1
  5. 2026Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesiscited by 0
  6. 2026LLM and Human Modes of Representationcited by 0
  7. 2026surprisal is Not a Theorycited by 0
  8. 2026On the use of foundation models in cognitive sciencecited by 0
  9. 2026CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognitioncited by 0
  10. 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0
  11. 2026Using LMs to Model the Effects of Context and Coreference during Sentence Comprehensioncited by 0