# A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing

arXiv: https://arxiv.org/abs/2310.16142 · HTML: https://mlsanity.com/paper/2310.16142/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 50 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans](https://mlsanity.com/paper/2106.03232/) (2021)
- [Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities](https://mlsanity.com/paper/2210.12187/) (2022)
- [Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention](https://mlsanity.com/paper/2104.12874/) (2021)
- [Single Headed Attention RNN: Stop Thinking With Your Head](https://mlsanity.com/paper/1911.11423/) (2019)
- [Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal](https://mlsanity.com/paper/2212.11185/) (2022)
- [Exploring the Syntactic Abilities of RNNs with Multi-task Learning](https://mlsanity.com/paper/1706.03542/) (2017)

## Built on by (most cited first, 22 of 22)

- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [Information Locality as an Inductive Bias for Neural Language Models](https://mlsanity.com/paper/2506.05136/) (2025)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Psychometric Predictive Power of Large Language Models](https://mlsanity.com/paper/2311.07484/) (2023)
- [If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?](https://mlsanity.com/paper/2502.11469/) (2025)
- [Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models](https://mlsanity.com/paper/2508.05803/) (2025)
- [What Makes Language Models Good-enough?](https://mlsanity.com/paper/2406.03666/) (2024)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)
- [Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects](https://mlsanity.com/paper/2603.16574/) (2026)
- [Memory Retrieval in Transformers: Insights from The Encoding Specificity Principle](https://mlsanity.com/paper/2601.20282/) (2026)
- [Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity](https://mlsanity.com/paper/2604.20789/) (2026)
- [A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models](https://mlsanity.com/paper/2604.26351/) (2026)
- [Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis](https://mlsanity.com/paper/2605.15440/) (2026)
- [Quantifying the cross-linguistic effects of syncretism on agreement attraction](https://mlsanity.com/paper/2605.21403/) (2026)
- [Surprisal Theory is Tautological (without Rational Grounding)](https://mlsanity.com/paper/2607.21574/) (2026)
- [Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?](https://mlsanity.com/paper/2608.23818/) (2026)
- [Factors Influencing the Emergence of Dependency Length Minimization in Neural Agent Simulations](https://mlsanity.com/paper/2609.06025/) (2026)
- [Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension](https://mlsanity.com/paper/2609.32119/) (2026)

## Abstract (arXiv)

Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While Ryu and Lewis show that attention patterns in specialized attention heads of GPT-2 are consistent with similarity-based interference, a key prediction of cue-based retrieval models, their method requires identifying syntactically specialized attention heads, and makes the cognitively implausible assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model's single attention head captures semantic and syntactic interference effects observed in human experiments.