mlsanityData to 14 Oct 2026

Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?

arXiv 2608.23818 ↗First posted 24 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers' attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 56 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

16 of 56 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  4. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  5. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  6. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  7. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  8. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  9. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  10. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 27
  11. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  12. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
4 more
  1. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
  2. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  3. 2025Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisitioncited by 10
  4. 2026Dual Alignment Between Language Model Layers and Human Sentence Processingcited by 2

Built on by

Not cited yet.