mlsanityData to 13 Oct 2026

Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

arXiv 2212.12131 ↗First posted 23 Dec 2022cs.CLcited by 112

Not on the map. Why?

Abstract (arXiv)

This work presents a detailed linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. First, regression analyses show a strictly monotonic, positive log-linear relationship between perplexity and fit to reading times for the more recently released five GPT-Neo variants and eight OPT variants on two separate datasets, replicating earlier results limited to just GPT-2 (Oh et al., 2022). Subsequently, analysis of residual errors reveals a systematic deviation of the larger variants, such as underpredicting reading times of named entities and making compensatory overpredictions for reading times of function words such as modals and conjunctions. These results suggest that the propensity of larger Transformer-based models to 'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations, which warrants caution in using pre-trained language models to study human language processing.

Key facts. Cited by 112 papers in the map (data through 13 Oct 2026). Builds on 20 of its 50 references that are in the map. First posted on arXiv 23 Dec 2022.

Builds on

20 of 50 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,024
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 118
  10. 2016Recurrent Neural Network Grammarscited by 109
  11. 2021Lower Perplexity is Not Always Human-Likecited by 52
  12. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
8 more
  1. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  2. 2017The Natural Stories Corpuscited by 40
  3. 2022Context Limitations Make Neural Language Models More Human-Likecited by 33
  4. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  5. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  6. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  7. 2021Relative Importance in Sentence Processingcited by 22
  8. 2022Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisalcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 15; 2024: 27; 2025: 26; 2026: 44.

112 papers.

Sort:

By area:

  1. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 76
  2. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  3. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 40
  4. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 39
  5. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 38
  6. 2024How to Compute the Probability of a Wordcited by 36
  7. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 35
  8. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  9. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  10. 2025From Language to Cognition: How LLMs Outgrow the Human Language Networkcited by 30
  11. 2025Large Language Models Are Human-Like Internallycited by 29
  12. 2024Generalized Measures of Anticipation and Responsivity in Online Language Processingcited by 23
38 more
  1. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  2. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  3. 2024On the Role of Context in Reading Time Predictioncited by 18
  4. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  5. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  6. 2024A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Rolescited by 14
  7. 2023Humans and language models diverge when predicting repeating textcited by 13
  8. 2023Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?cited by 13
  9. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  10. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  11. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  12. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  13. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  14. 2024What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Lengthcited by 10
  15. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  16. 2024Towards a Similarity-adjusted Surprisal Theorycited by 9
  17. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  18. 2023Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attentioncited by 7
  19. 2023Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?cited by 7
  20. 2024Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metricscited by 6
  21. 2024SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Modelscited by 6
  22. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 6
  23. 2024EMTeC: A Corpus of Eye Movements on Machine-Generated Textscited by 5
  24. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 5
  25. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  26. 2026Post-training makes large language models less human-likecited by 5
  27. 2023Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosodycited by 4
  28. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  29. 2023Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialoguecited by 4
  30. 2024Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Modelscited by 4
  31. 2025The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakagecited by 4
  32. 2024Brain-Like Language Processing via a Shallow Untrained Multihead Attention Networkcited by 3
  33. 2025Signatures of human-like processing in Transformer forward passescited by 3
  34. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3
  35. 2024Computational Sentence-level Metrics Predicting Human Sentence Comprehensioncited by 2
  36. 2024Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differencescited by 2
  37. 2025Strategic resource allocation in memory encoding: An efficiency principle shaping language processingcited by 2
  38. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2

50 most cited of 112 All 112 (JSON)