mlsanityData to 7 Oct 2026

Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities

arXiv 2210.12187 ↗First posted 21 Oct 2022cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Humans exhibit garden path effects: When reading sentences that are temporarily structurally ambiguous, they slow down when the structure is disambiguated in favor of the less preferred alternative. Surprisal theory (Hale, 2001; Levy, 2008), a prominent explanation of this finding, proposes that these slowdowns are due to the unpredictability of each of the words that occur in these sentences. Challenging this hypothesis, van Schijndel & Linzen (2021) find that estimates of the cost of word predictability derived from language models severely underestimate the magnitude of human garden path effects. In this work, we consider whether this underestimation is due to the fact that humans weight syntactic factors in their predictions more highly than language models do. We propose a method for estimating syntactic predictability from a language model, allowing us to weigh the cost of lexical and syntactic predictability independently. We find that treating syntactic predictability independently from lexical predictability indeed results in larger estimates of garden path. At the same time, even when syntactic predictability is independently weighted, surprisal still greatly underestimate the magnitude of human garden path effects. Our results support the hypothesis that predictability is not the only factor responsible for the processing cost associated with garden path sentences.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 4 of its 25 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

4 of 25 references.

References · 4

  1. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  2. 2016Recurrent Neural Network Grammarscited by 109
  3. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  4. 2017Exploring the Syntactic Abilities of RNNs with Multi-task Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 10; 2025: 4; 2026: 8.

30 papers.

Sort:

By area:

  1. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  2. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  3. 2023Dissociating language and thought in large language modelscited by 27
  4. 2024Generalized Measures of Anticipation and Responsivity in Online Language Processingcited by 23
  5. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  6. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  7. 2024A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Rolescited by 14
  8. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  9. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  10. 2023Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semanticscited by 11
  11. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  12. 2025To model human linguistic prediction, make LLMs less superhumancited by 11
18 more
  1. 2024Towards a Similarity-adjusted Surprisal Theorycited by 9
  2. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  3. 2024Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Modelscited by 4
  4. 2023Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativizationcited by 2
  5. 2024Computational Sentence-level Metrics Predicting Human Sentence Comprehensioncited by 2
  6. 2025Signatures of human-like processing in Transformer forward passescited by 2
  7. 2026Dual Alignment Between Language Model Layers and Human Sentence Processingcited by 2
  8. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
  9. 2024Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processescited by 1
  10. 2024When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementalitycited by 1
  11. 2025Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorlycited by 1
  12. 2026Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do notcited by 1
  13. 2026An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisalcited by 1
  14. 2025The Challenge and Reward of Fair Play in Narrative: A Computational Approachcited by 0
  15. 2026Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesiscited by 0
  16. 2026LaViSA: A Language and Vision Structural Ambiguity Benchmarkcited by 0
  17. 2026Syntactic Belief Update as the Driver of Garden Path Processing Difficultycited by 0
  18. 2026Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Modelscited by 0