mlsanityData to 14 Oct 2026

A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles

arXiv 2410.16139 ↗First posted 21 Oct 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We present a systematic evaluation of large language models' sensitivity to argument roles, i.e., who did what to whom, by replicating psycholinguistic studies on human argument role processing. In three experiments, we find that language models are able to distinguish verbs that appear in plausible and implausible contexts, where plausibility is determined through the relation between the verb and its preceding arguments. However, none of the models capture the same selective patterns that human comprehenders exhibit during real-time verb prediction. This indicates that language models' capacity to detect verb plausibility does not arise from the same mechanism that underlies human real-time sentence processing.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

31 of 57 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  11. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
19 more
  1. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  2. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  3. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  4. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  5. 2020Syntactic Structure from Deep Learningcited by 99
  6. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  7. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  8. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  9. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
  10. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  11. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  12. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 27
  13. 2021How is BERT surprised? Layerwise detection of linguistic anomaliescited by 24
  14. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  15. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  16. 2023How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structurecited by 13
  17. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  18. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  19. 2022Syntactic Substitutability as Unsupervised Dependency Syntaxcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2025A Review of Incorporating Psychological Theories in LLMscited by 6
  3. 2025How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMscited by 3
  4. 2025From Monolingual to Bilingual: Investigating Language Conditioning in Large Language Models for Psycholinguistic Taskscited by 1
  5. 2025Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disordercited by 0
  6. 2025Do Large Language Models know who did what to whom?cited by 0
  7. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  8. 2025Large language models have learned to use languagecited by 0
  9. 2025Large language models are not about natural languagecited by 0
  10. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  11. 2026Are Language Models Models?cited by 0
  12. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
2 more
  1. 2026You Can't Fight in Here! This is BBS!cited by 0
  2. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0