mlsanityData to 14 Oct 2026

Language Models Grow Less Humanlike beyond Phase Transition

arXiv 2502.18802 ↗First posted 26 Feb 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

LMs' alignment with human reading behavior (i.e. psychometric predictive power; PPP) is known to improve during pretraining up to a tipping point, beyond which it either plateaus or degrades. Various factors, such as word frequency, recency bias in attention, and context size, have been theorized to affect PPP, yet there is no current account that explains why such a tipping point exists, and how it interacts with LMs' pretraining dynamics more generally. We hypothesize that the underlying factor is a pretraining phase transition, characterized by the rapid emergence of specialized attention heads. We conduct a series of correlational and causal experiments to show that such a phase transition is responsible for the tipping point in PPP. We then show that, rather than producing attention patterns that contribute to the degradation in PPP, phase transitions alter the subsequent learning dynamics of the model, such that further training keeps damaging PPP.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 49 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

24 of 49 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2022In-context Learning and Induction Headscited by 964
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  12. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
12 more
  1. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  2. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  3. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  4. 2021Revisiting the Uniform Information Density Hypothesiscited by 68
  5. 2021Lower Perplexity is Not Always Human-Likecited by 52
  6. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  7. 2017The Natural Stories Corpuscited by 40
  8. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  9. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  10. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  11. 2024On the Role of Context in Reading Time Predictioncited by 20
  12. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 7
  2. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  3. 2025Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language modelscited by 3
  4. 2026Transformers perform adaptive partial poolingcited by 3
  5. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  6. 2026Function Words as Statistical Cues for Language Learningcited by 1
  7. 2025Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attractioncited by 0
  8. 2026Phase Transitions in Affective Meaning Divergence: The Hidden Drift Before the Breakcited by 0
  9. 2026Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Modelscited by 0
  10. 2026Beetle: A Bilingual Model Suite for Modelling Second-Language Processingcited by 0