mlsanityData to 14 Oct 2026

N-gram-like Language Models Predict Naturalistic Reading Time Best

arXiv 2603.09872 ↗First posted 10 Mar 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent work has found that contemporary language models such as transformers can become so good at next-word prediction that the probabilities they calculate become worse for predicting naturalistic reading time. In this paper, we propose that this can be explained by reading time being shaped by simple n-gram statistics rather than the more complex statistics learned by state-of-the-art transformer language models. We demonstrate that the neural language models whose predictions are most correlated with n-gram probability are also those that calculate probabilities that are the most correlated with eye-tracking-based metrics of reading time on naturalistic text.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 90 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

34 of 90 references.

References · 34

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  5. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  6. 2015Visualizing and Understanding Recurrent Networkscited by 160
  7. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  8. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  9. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  10. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  11. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  12. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
22 more
  1. 2021Revisiting the Uniform Information Density Hypothesiscited by 68
  2. 2021Lower Perplexity is Not Always Human-Likecited by 52
  3. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  4. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  5. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  6. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  7. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 35
  8. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  9. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 27
  10. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  11. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  12. 2024On the Role of Context in Reading Time Predictioncited by 20
  13. 2021So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgementscited by 15
  14. 2021Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?cited by 14
  15. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  16. 2023Humans and language models diverge when predicting repeating textcited by 13
  17. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  18. 2024Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metricscited by 6
  19. 2025Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Modelscited by 5
  20. 2025The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakagecited by 5
  21. 2022Implicit N-grams Induced by Recurrencecited by 4
  22. 2021Local word statistics affect reading times independently of surprisalcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Energy-Based Transformers as Predictors of Reading Difficultycited by 0
  2. 2026On the use of foundation models in cognitive sciencecited by 0