# Tracking Naturalistic Linguistic Predictions with Deep Neural Language Models

arXiv: https://arxiv.org/abs/1909.04400 · HTML: https://mlsanity.com/paper/1909.04400/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures](https://mlsanity.com/paper/2311.04547/) (2023)
- [Long-range and hierarchical language predictions in brains and algorithms](https://mlsanity.com/paper/2111.14232/) (2021)
- [MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing](https://mlsanity.com/paper/2208.11488/) (2022)
- [Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting](https://mlsanity.com/paper/2405.20101/) (2024)
- [YNote: A Novel Music Notation for Fine-Tuning LLMs in Music Generation](https://mlsanity.com/paper/2502.10467/) (2025)
- [Comparing Computational Architectures for Automated Journalism](https://mlsanity.com/paper/2210.04107/) (2022)
- [BLAB Reporter: Automated journalism covering the Blue Amazon](https://mlsanity.com/paper/2210.06431/) (2022)
- [Can We Locate and Prevent Stereotypes in LLMs?](https://mlsanity.com/paper/2604.19764/) (2026)
- [Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models](https://mlsanity.com/paper/2607.16549/) (2026)

## Abstract (arXiv)

Prediction in language has traditionally been studied using simple designs in which neural responses to expected and unexpected words are compared in a categorical fashion. However, these designs have been contested as being `prediction encouraging', potentially exaggerating the importance of prediction in language understanding. A few recent studies have begun to address these worries by using model-based approaches to probe the effects of linguistic predictability in naturalistic stimuli (e.g. continuous narrative). However, these studies so far only looked at very local forms of prediction, using models that take no more than the prior two words into account when computing a word's predictability. Here, we extend this approach using a state-of-the-art neural language model that can take roughly 500 times longer linguistic contexts into account. Predictability estimates from the neural network offer a much better fit to EEG data from subjects listening to naturalistic narrative than simpler models, and reveal strong surprise responses akin to the P200 and N400. These results show that predictability effects in language are not a side-effect of simple designs, and demonstrate the practical use of recent advances in AI for the cognitive neuroscience of language.