mlsanityData to 7 Oct 2026

Large Language Models Are Human-Like Internally

arXiv 2502.01615 ↗First posted 3 Feb 2025cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Recent cognitive modeling studies have reported that larger language models (LMs) exhibit a poorer fit to human reading behavior (Oh and Schuler, 2023b; Shain et al., 2024; Kuribayashi et al., 2024), leading to claims of their cognitive implausibility. In this paper, we revisit this argument through the lens of mechanistic interpretability and argue that prior conclusions were skewed by an exclusive focus on the final layers of LMs. Our analysis reveals that next-word probabilities derived from internal layers of larger LMs align with human sentence processing data as well as, or better than, those from smaller LMs. This alignment holds consistently across behavioral (self-paced reading times, gaze durations, MAZE task processing times) and neurophysiological (N400 brain potentials) measures, challenging earlier mixed results and suggesting that the cognitive plausibility of larger LMs has been underestimated. Furthermore, we first identify an intriguing relationship between LM layers and human measures: earlier layers correspond more closely with fast gaze durations, while later layers better align with relatively slower signals such as N400 potentials and MAZE processing times. Our work opens new avenues for interdisciplinary research at the intersection of mechanistic interpretability and cognitive modeling.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 43 of its 103 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

43 of 103 references.

References · 43

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  7. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  8. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  10. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  11. 2021PonderNet: Learning to Pondercited by 156
  12. 2022Analyzing Transformers in Embedding Spacecited by 131
31 more
  1. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  2. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  3. 2019On Identifiability in Transformerscited by 115
  4. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  5. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  6. 2016Recurrent Neural Network Grammarscited by 109
  7. 2021Lower Perplexity is Not Always Human-Likecited by 52
  8. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  9. 2024Language models align with human judgments on key grammatical constructionscited by 49
  10. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  11. 2017The Natural Stories Corpuscited by 40
  12. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  13. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 39
  14. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  15. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  16. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  17. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  18. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  19. 2024Generalized Measures of Anticipation and Responsivity in Online Language Processingcited by 23
  20. 2022On the Effect of Anticipation on Reading Timescited by 18
  21. 2019Towards a Quantum-Like Cognitive Architecture for Decision-Makingcited by 17
  22. 2024On the Role of Context in Reading Time Predictioncited by 17
  23. 2023Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?cited by 13
  24. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  25. 2022Analyzing Wrap-Up Effects through an Information-Theoretic Lenscited by 10
  26. 2024Towards a Similarity-adjusted Surprisal Theorycited by 9
  27. 2014Tracking Dynamic Point Processes on Networkscited by 8
  28. 2024Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metricscited by 6
  29. 2021Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammarscited by 5
  30. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  31. 2024SPAWNing Structural Priming Predictions from a Cognitively Motivated Parsercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 17.

28 papers.

Sort:

By area:

  1. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  2. 2025Agentic Knowledgeable Self-awarenesscited by 8
  3. 2025Diversity Boosts AI-Generated Text Detectioncited by 6
  4. 2026Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judgecited by 3
  5. 2026When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalitiescited by 3
  6. 2025Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibilitycited by 2
  7. 2025Signatures of human-like processing in Transformer forward passescited by 2
  8. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2
  9. 2025Comparing Human and Language Models Sentence Processing Difficulties on Complex Structurescited by 2
  10. 2025Metacognitive Sensitivity for Test-Time Dynamic Model Selectioncited by 2
  11. 2026Dual Alignment Between Language Model Layers and Human Sentence Processingcited by 2
  12. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
16 more
  1. 2025Vectors from Larger Language Models Predict Human Reading Time and fMRI Data More Poorly when Dimensionality Expansion is Controlledcited by 1
  2. 2025AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychologycited by 1
  3. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  4. 2025In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violationscited by 0
  5. 2026CBMAS: Cognitive Behavioral Modeling via Activation Steeringcited by 0
  6. 2026Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and Englishcited by 0
  7. 2026Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMscited by 0
  8. 2026Probing for Reading Timescited by 0
  9. 2026When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMscited by 0
  10. 2026Timesteps of Mamba Align with Human Reading Timescited by 0
  11. 2026Production and Perception in LLMs: A Token Probability Approachcited by 0
  12. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0
  13. 2026On the use of foundation models in cognitive sciencecited by 0
  14. 2026Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Modelscited by 0
  15. 2026Using LMs to Model the Effects of Context and Coreference during Sentence Comprehensioncited by 0
  16. 2026Reader Proficiency Shapes Layer-wise Surprisal Profilescited by 0