# Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities

arXiv: https://arxiv.org/abs/2210.12187 · HTML: https://mlsanity.com/paper/2210.12187/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 4 of 4)

- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans](https://mlsanity.com/paper/2106.03232/) (2021)
- [Exploring the Syntactic Abilities of RNNs with Multi-task Learning](https://mlsanity.com/paper/1706.03542/) (2017)

## Built on by (most cited first, 30 of 30)

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence](https://mlsanity.com/paper/2405.15750/) (2024)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives](https://mlsanity.com/paper/2310.13676/) (2023)
- [A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://mlsanity.com/paper/2310.16142/) (2023)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods](https://mlsanity.com/paper/2403.00998/) (2024)
- [When Language Models Fall in Love: Animacy Processing in Transformer Language Models](https://mlsanity.com/paper/2310.15004/) (2023)
- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures](https://mlsanity.com/paper/2311.04547/) (2023)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models](https://mlsanity.com/paper/2412.05353/) (2024)
- [Signatures of human-like processing in Transformer forward passes](https://mlsanity.com/paper/2504.14107/) (2025)
- [Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization](https://mlsanity.com/paper/2305.14917/) (2023)
- [Computational Sentence-level Metrics Predicting Human Sentence Comprehension](https://mlsanity.com/paper/2403.15822/) (2024)
- [Dual Alignment Between Language Model Layers and Human Sentence Processing](https://mlsanity.com/paper/2604.18563/) (2026)
- [On the Proper Treatment of Units in Surprisal Theory](https://mlsanity.com/paper/2604.28147/) (2026)
- [Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes](https://mlsanity.com/paper/2402.01352/) (2024)
- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)
- [Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly](https://mlsanity.com/paper/2506.11338/) (2025)
- [Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not](https://mlsanity.com/paper/2604.04825/) (2026)
- [An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal](https://mlsanity.com/paper/2604.18293/) (2026)
- [The Challenge and Reward of Fair Play in Narrative: A Computational Approach](https://mlsanity.com/paper/2507.13841/) (2025)
- [Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis](https://mlsanity.com/paper/2605.15440/) (2026)
- [LaViSA: A Language and Vision Structural Ambiguity Benchmark](https://mlsanity.com/paper/2606.19552/) (2026)
- [Syntactic Belief Update as the Driver of Garden Path Processing Difficulty](https://mlsanity.com/paper/2606.27206/) (2026)
- [Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models](https://mlsanity.com/paper/2607.16549/) (2026)

## Abstract (arXiv)

Humans exhibit garden path effects: When reading sentences that are temporarily structurally ambiguous, they slow down when the structure is disambiguated in favor of the less preferred alternative. Surprisal theory (Hale, 2001; Levy, 2008), a prominent explanation of this finding, proposes that these slowdowns are due to the unpredictability of each of the words that occur in these sentences. Challenging this hypothesis, van Schijndel & Linzen (2021) find that estimates of the cost of word predictability derived from language models severely underestimate the magnitude of human garden path effects. In this work, we consider whether this underestimation is due to the fact that humans weight syntactic factors in their predictions more highly than language models do. We propose a method for estimating syntactic predictability from a language model, allowing us to weigh the cost of lexical and syntactic predictability independently. We find that treating syntactic predictability independently from lexical predictability indeed results in larger estimates of garden path. At the same time, even when syntactic predictability is independently weighted, surprisal still greatly underestimate the magnitude of human garden path effects. Our results support the hypothesis that predictability is not the only factor responsible for the processing cost associated with garden path sentences.