mlsanityData to 7 Oct 2026

Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

arXiv 2404.19178 ↗First posted 30 Apr 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match - and in some cases, exceed - the performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 114 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

29 of 114 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  12. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
17 more
  1. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  2. 2015A Primer on Neural Network Models for Natural Language Processingcited by 75
  3. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  4. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  5. 2021Lower Perplexity is Not Always Human-Likecited by 52
  6. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  7. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  8. 2017The Natural Stories Corpuscited by 40
  9. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  10. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  11. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  12. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  13. 2021So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgementscited by 15
  14. 2021Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?cited by 14
  15. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  16. 2023Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semanticscited by 11
  17. 2022Collateral facilitation in humans and language modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025Large Language Models Are Human-Like Internallycited by 28
  2. 2024A Survey of RWKVcited by 5
  3. 2025Signatures of human-like processing in Transformer forward passescited by 2
  4. 2026LLM surprisal is necessary but not sufficient to capture English garden-path effects: Evidence from joint latent modeling of reading paradigmscited by 1
  5. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  6. 2026What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  2. 2024Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer
  3. 2025Modeling Language as a Sequence of Thoughts
  4. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  5. 2024A critical review of methods and challenges in large language models
  6. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  7. 2024Does Transformer Interpretability Transfer to RNNs?
  8. 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI