# Testing the Predictions of Surprisal Theory in 11 Languages

arXiv: https://arxiv.org/abs/2307.03667 · HTML: https://mlsanity.com/paper/2307.03667/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 75 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [Is Multilingual BERT Fluent in Language Generation?](https://mlsanity.com/paper/1910.03806/) (2019)
- [Give your Text Representation Models some Love: the Case for Basque](https://mlsanity.com/paper/2004.00033/) (2020)
- [A surprisal--duration trade-off across and within the world's languages](https://mlsanity.com/paper/2109.15000/) (2021)
- [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022)

## Built on by (most cited first, 30 of 77)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [How to Compute the Probability of a Word](https://mlsanity.com/paper/2406.14561/) (2024)
- [Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities](https://mlsanity.com/paper/2406.10851/) (2024)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?](https://mlsanity.com/paper/2310.17774/) (2023)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Language Models Grow Less Humanlike beyond Phase Transition](https://mlsanity.com/paper/2502.18802/) (2025)
- [MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models](https://mlsanity.com/paper/2403.01116/) (2024)
- [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024)
- [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025)
- [Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?](https://mlsanity.com/paper/2311.09325/) (2023)
- [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024)
- [Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?](https://mlsanity.com/paper/2410.03240/) (2024)
- [The Impact of Token Granularity on the Predictive Power of Language Model Surprisal](https://mlsanity.com/paper/2412.11940/) (2024)
- [The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage](https://mlsanity.com/paper/2506.01172/) (2025)
- [Psychometric Predictive Power of Large Language Models](https://mlsanity.com/paper/2311.07484/) (2023)
- [How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?](https://mlsanity.com/paper/2507.22209/) (2025)
- [Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs](https://mlsanity.com/paper/2502.19078/) (2025)
- [Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent](https://mlsanity.com/paper/2505.07659/) (2025)
- [A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior](https://mlsanity.com/paper/2506.19999/) (2025)
- [Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings](https://mlsanity.com/paper/2509.14405/) (2025)
- [The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices](https://mlsanity.com/paper/2603.18482/) (2026)
- [Computational Sentence-level Metrics Predicting Human Sentence Comprehension](https://mlsanity.com/paper/2403.15822/) (2024)
- [Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences](https://mlsanity.com/paper/2406.04988/) (2024)
- [Decomposition of surprisal: Unified computational model of ERP components in language processing](https://mlsanity.com/paper/2409.06803/) (2024)

## Abstract (arXiv)

A fundamental result in psycholinguistics is that less predictable words take a longer time to process. One theoretical explanation for this finding is Surprisal Theory (Hale, 2001; Levy, 2008), which quantifies a word's predictability as its surprisal, i.e. its negative log-probability given a context. While evidence supporting the predictions of Surprisal Theory have been replicated widely, most have focused on a very narrow slice of data: native English speakers reading English texts. Indeed, no comprehensive multilingual analysis exists. We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families. Deriving estimates from language models trained on monolingual and multilingual corpora, we test three predictions associated with surprisal theory: (i) whether surprisal is predictive of reading times; (ii) whether expected surprisal, i.e. contextual entropy, is predictive of reading times; (iii) and whether the linking function between surprisal and reading times is linear. We find that all three predictions are borne out crosslinguistically. By focusing on a more diverse set of languages, we argue that these results offer the most robust link to-date between information theory and incremental language processing across languages.