mlsanityData to 7 Oct 2026

Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension

arXiv 2609.32119 ↗First posted 26 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs' context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans' reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (< 20 tokens) successfully capture local memory limitations, expanded contexts (500--1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 82 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

21 of 82 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  7. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  8. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  9. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  10. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  11. 2017The Natural Stories Corpuscited by 40
  12. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
9 more
  1. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  2. 2025Large Language Models Are Human-Like Internallycited by 28
  3. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  4. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  5. 2024Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discoursecited by 14
  6. 2024eRST: A Signaled Graph Theory of Discourse Relations and Organizationcited by 13
  7. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  8. 2025The Harmonic Structure of Information Contourscited by 7
  9. 2026Syntactically-guided Information Maintenance in Sentence Comprehensioncited by 1

Built on by

Not cited yet.