# Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?

arXiv: https://arxiv.org/abs/2608.23818 · HTML: https://mlsanity.com/paper/2608.23818/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Forgetting Transformer: Softmax Attention with a Forget Gate](https://mlsanity.com/paper/2503.02130/) (2025)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention](https://mlsanity.com/paper/2104.12874/) (2021)
- [A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://mlsanity.com/paper/2310.16142/) (2023)
- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study](https://mlsanity.com/paper/2410.17980/) (2024)
- [Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures](https://mlsanity.com/paper/2311.04547/) (2023)
- [Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition](https://mlsanity.com/paper/2502.04795/) (2025)
- [Dual Alignment Between Language Model Layers and Human Sentence Processing](https://mlsanity.com/paper/2604.18563/) (2026)

## Abstract (arXiv)

Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers' attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.