mlsanityData to 7 Oct 2026

Linear Recency Bias During Training Improves Transformers' Fit to Reading Times

arXiv 2409.11250 ↗First posted 17 Sep 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent psycholinguistic research has compared human reading times to surprisal estimates from language models to study the factors shaping human sentence processing difficulty. Previous studies have shown a strong fit between surprisal values from Transformers and reading times. However, standard Transformers work with a lossless representation of the entire previous linguistic context, unlike models of human language processing that include memory decay. To bridge this gap, this paper evaluates a modification of the Transformer model that uses ALiBi (Press et al., 2022), a recency bias added to attention scores. Surprisal estimates with ALiBi show an improved fit to human reading times compared to a standard Transformer baseline. A subsequent analysis of attention heads suggests that ALiBi's mixture of slopes -- which determine the rate of memory decay in each attention head -- may play a role in the improvement by helping models with ALiBi to track different kinds of linguistic dependencies.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 20 of its 55 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

20 of 55 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  10. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  11. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  12. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
8 more
  1. 2017The Natural Stories Corpuscited by 40
  2. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  3. 2024How to Compute the Probability of a Wordcited by 36
  4. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  5. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  6. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  7. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  8. 2022Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisalcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

  1. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
  2. 2025Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisitioncited by 10
  3. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  4. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3
  5. 2025What is the Best Sequence Length for BABYLM?cited by 2
  6. 2026Simulating Human Memory with Language Modelscited by 2
  7. 2025Analysis of LLM as a grammatical feature tagger for African American Englishcited by 1
  8. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  9. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  10. 2026Working Memory Constraints Scaffold Learning in Transformers under Data Scarcitycited by 0
  11. 2026Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesiscited by 0
  12. 2026Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?cited by 0
2 more
  1. 2026Factors Influencing the Emergence of Dependency Length Minimization in Neural Agent Simulationscited by 0
  2. 2026Using LMs to Model the Effects of Context and Coreference during Sentence Comprehensioncited by 0