mlsanityData to 7 Oct 2026

A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing

arXiv 2310.16142 ↗First posted 24 Oct 2023cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While Ryu and Lewis show that attention patterns in specialized attention heads of GPT-2 are consistent with similarity-based interference, a key prediction of cue-based retrieval models, their method requires identifying syntactically specialized attention heads, and makes the cognitively implausible assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model's single attention head captures semantic and syntactic interference effects observed in human experiments.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 15 of its 50 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

15 of 50 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  6. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  7. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  8. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  9. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  10. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  11. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  12. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
3 more
  1. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  2. 2022Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisalcited by 14
  3. 2017Exploring the Syntactic Abilities of RNNs with Multi-task Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 6; 2026: 11.

22 papers.

Sort:

By area:

  1. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  2. 2024A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Rolescited by 14
  3. 2025Information Locality as an Inductive Bias for Neural Language Modelscited by 14
  4. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  5. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  6. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  7. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  8. 2025If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?cited by 4
  9. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3
  10. 2024What Makes Language Models Good-enough?cited by 2
  11. 2025Strategic resource allocation in memory encoding: An efficiency principle shaping language processingcited by 2
  12. 2026Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effectscited by 2
10 more
  1. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  2. 2026Memory Retrieval in Transformers: Insights from The Encoding Specificity Principlecited by 0
  3. 2026Working Memory Constraints Scaffold Learning in Transformers under Data Scarcitycited by 0
  4. 2026A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Modelscited by 0
  5. 2026Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesiscited by 0
  6. 2026Quantifying the cross-linguistic effects of syncretism on agreement attractioncited by 0
  7. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0
  8. 2026Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?cited by 0
  9. 2026Factors Influencing the Emergence of Dependency Length Minimization in Neural Agent Simulationscited by 0
  10. 2026Using LMs to Model the Effects of Context and Coreference during Sentence Comprehensioncited by 0