mlsanityData to 7 Oct 2026

Context Limitations Make Neural Language Models More Human-Like

arXiv 2205.11463 ↗First posted 23 May 2022cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) have been used in cognitive modeling as well as engineering studies -- they compute information-theoretic complexity metrics that simulate humans' cognitive load during reading. This study highlights a limitation of modern neural LMs as the model of choice for this purpose: there is a discrepancy between their context access capacities and that of humans. Our results showed that constraining the LMs' context access improved their simulation of human reading behavior. We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs' context access might enhance their cognitive plausibility.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 19 of its 64 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

19 of 64 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  6. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  7. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  8. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  9. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
  10. 2021Lower Perplexity is Not Always Human-Likecited by 52
  11. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  12. 2021What Context Features Can Transformer Language Models Use?cited by 49
7 more
  1. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  2. 2019Linking artificial and human neural representations of languagecited by 37
  3. 2020"LazImpa": Lazy and Impatient neural agents learn to communicate efficientlycited by 23
  4. 2022Emergent Communication for Understanding Human Language Evolution: What's Missing?cited by 23
  5. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  6. 2021Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?cited by 14
  7. 2021Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammarscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 5; 2025: 6; 2026: 11.

32 papers.

Sort:

By area:

  1. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  2. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  3. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  4. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  5. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  6. 2025Large Language Models Are Human-Like Internallycited by 28
  7. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  8. 2022On the Effect of Anticipation on Reading Timescited by 18
  9. 2024On the Role of Context in Reading Time Predictioncited by 17
  10. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  11. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  12. 2023Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segmentscited by 11
20 more
  1. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  2. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  3. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  4. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  5. 2023Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?cited by 7
  6. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  7. 2025The time scale of redundancy between prosody and linguistic contextcited by 4
  8. 2026Dual Alignment Between Language Model Layers and Human Sentence Processingcited by 2
  9. 2025Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorlycited by 1
  10. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  11. 2026An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisalcited by 1
  12. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  13. 2026Working Memory Constraints Scaffold Learning in Transformers under Data Scarcitycited by 0
  14. 2026A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Modelscited by 0
  15. 2026Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Readingcited by 0
  16. 2026Timesteps of Mamba Align with Human Reading Timescited by 0
  17. 2026Production and Perception in LLMs: A Token Probability Approachcited by 0
  18. 2026Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?cited by 0
  19. 2026Using LMs to Model the Effects of Context and Coreference during Sentence Comprehensioncited by 0
  20. 2026Reader Proficiency Shapes Layer-wise Surprisal Profilescited by 0