# Human Sentence Processing: Recurrence or Attention?

arXiv: https://arxiv.org/abs/2005.09471 · HTML: https://mlsanity.com/paper/2005.09471/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 30 of 51)

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://mlsanity.com/paper/2304.11389/) (2023)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading](https://mlsanity.com/paper/2304.10784/) (2023)
- [So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements](https://mlsanity.com/paper/2109.01226/) (2021)
- [Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal](https://mlsanity.com/paper/2212.11185/) (2022)
- [ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts](https://mlsanity.com/paper/2310.15587/) (2023)
- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [On the Proper Treatment of Tokenization in Psycholinguistics](https://mlsanity.com/paper/2410.02691/) (2024)
- [When Language Models Fall in Love: Animacy Processing in Transformer Language Models](https://mlsanity.com/paper/2310.15004/) (2023)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models](https://mlsanity.com/paper/2210.09819/) (2022)
- [ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization](https://mlsanity.com/paper/2506.08712/) (2025)
- [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://mlsanity.com/paper/2304.04758/) (2023)
- [Can Transformer Language Models Predict Psychometric Properties?](https://mlsanity.com/paper/2106.06849/) (2021)
- [Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns](https://mlsanity.com/paper/2203.08085/) (2022)
- [Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition](https://mlsanity.com/paper/2203.13285/) (2022)
- [Testing the limits of natural language models for predicting human language judgments](https://mlsanity.com/paper/2204.03592/) (2022)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)
- [Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?](https://mlsanity.com/paper/2311.09325/) (2023)
- [Can Peanuts Fall in Love with Distributional Semantics?](https://mlsanity.com/paper/2301.08731/) (2023)
- [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024)
- [Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars](https://mlsanity.com/paper/2109.04939/) (2021)
- [Predicting Human Psychometric Properties Using Computational Language Models](https://mlsanity.com/paper/2205.06203/) (2022)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have long been an architecture of interest for computational models of human sentence processing. The recently introduced Transformer architecture outperforms RNNs on many natural language processing tasks but little is known about its ability to model human language processing. We compare Transformer- and RNN-based language models' ability to account for measures of human reading effort. Our analysis shows Transformers to outperform RNNs in explaining self-paced reading times and neural activity during reading English sentences, challenging the widely held idea that human sentence processing involves recurrent and immediate processing and provides evidence for cue-based retrieval.