# Context Limitations Make Neural Language Models More Human-Like

arXiv: https://arxiv.org/abs/2205.11463 · HTML: https://mlsanity.com/paper/2205.11463/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 64 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [Human Sentence Processing: Recurrence or Attention?](https://mlsanity.com/paper/2005.09471/) (2020)
- [What Context Features Can Transformer Language Models Use?](https://mlsanity.com/paper/2106.08367/) (2021)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- ["LazImpa": Lazy and Impatient neural agents learn to communicate efficiently](https://mlsanity.com/paper/2010.01878/) (2020)
- [Emergent Communication for Understanding Human Language Evolution: What's Missing?](https://mlsanity.com/paper/2204.10590/) (2022)
- [Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling](https://mlsanity.com/paper/2009.03954/) (2020)
- [Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?](https://mlsanity.com/paper/2107.09648/) (2021)
- [Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars](https://mlsanity.com/paper/2109.04939/) (2021)

## Built on by (most cited first, 30 of 34)

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Can language models handle recursively nested grammatical structures? A case study on comparing models and humans](https://mlsanity.com/paper/2210.15303/) (2022)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://mlsanity.com/paper/2304.11389/) (2023)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://mlsanity.com/paper/2310.16142/) (2023)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022)
- [Generative Emergent Communication: Large Language Model is a Collective World Model](https://mlsanity.com/paper/2501.00226/) (2024)
- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segments](https://mlsanity.com/paper/2311.04453/) (2023)
- [Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures](https://mlsanity.com/paper/2311.04547/) (2023)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Language Models Grow Less Humanlike beyond Phase Transition](https://mlsanity.com/paper/2502.18802/) (2025)
- [Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?](https://mlsanity.com/paper/2311.09325/) (2023)
- [Psychometric Predictive Power of Large Language Models](https://mlsanity.com/paper/2311.07484/) (2023)
- [The time scale of redundancy between prosody and linguistic context](https://mlsanity.com/paper/2503.11630/) (2025)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)
- [Dual Alignment Between Language Model Layers and Human Sentence Processing](https://mlsanity.com/paper/2604.18563/) (2026)
- [Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly](https://mlsanity.com/paper/2506.11338/) (2025)
- [An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal](https://mlsanity.com/paper/2604.18293/) (2026)
- [Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking](https://mlsanity.com/paper/2507.08014/) (2025)
- [Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity](https://mlsanity.com/paper/2604.20789/) (2026)
- [A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models](https://mlsanity.com/paper/2604.26351/) (2026)
- [Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Reading](https://mlsanity.com/paper/2606.01964/) (2026)
- [Timesteps of Mamba Align with Human Reading Times](https://mlsanity.com/paper/2606.29904/) (2026)
- [Production and Perception in LLMs: A Token Probability Approach](https://mlsanity.com/paper/2607.11703/) (2026)
- [Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?](https://mlsanity.com/paper/2608.23818/) (2026)

## Abstract (arXiv)

Language models (LMs) have been used in cognitive modeling as well as engineering studies -- they compute information-theoretic complexity metrics that simulate humans' cognitive load during reading. This study highlights a limitation of modern neural LMs as the model of choice for this purpose: there is a discrepancy between their context access capacities and that of humans. Our results showed that constraining the LMs' context access improved their simulation of human reading behavior. We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs' context access might enhance their cognitive plausibility.