# Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling

arXiv: https://arxiv.org/abs/2009.03954 · HTML: https://mlsanity.com/paper/2009.03954/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2020.

## Built on by (most cited first, 22 of 22)

- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence](https://mlsanity.com/paper/2405.15750/) (2024)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [Humans and language models diverge when predicting repeating text](https://mlsanity.com/paper/2310.06408/) (2023)
- [Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?](https://mlsanity.com/paper/2310.17774/) (2023)
- [Can Transformer Language Models Predict Psychometric Properties?](https://mlsanity.com/paper/2106.06849/) (2021)
- [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024)
- [Predicting Human Psychometric Properties Using Computational Language Models](https://mlsanity.com/paper/2205.06203/) (2022)
- [Large Language Models for Psycholinguistic Plausibility Pretesting](https://mlsanity.com/paper/2402.05455/) (2024)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)
- [Memory in humans and deep language models: Linking hypotheses for model augmentation](https://mlsanity.com/paper/2210.01869/) (2022)
- [Composition, Attention, or Both?](https://mlsanity.com/paper/2210.12958/) (2022)
- [A Free Probabilistic Framework for Analyzing the Transformer-based Language Models](https://mlsanity.com/paper/2506.16550/) (2025)
- [Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal](https://mlsanity.com/paper/2601.09886/) (2026)
- [Can LLMs capture stable human-generated sentence entropy measures?](https://mlsanity.com/paper/2602.04570/) (2026)
- [Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures](https://mlsanity.com/paper/2305.18915/) (2023)
- [Generative Linguistics, Large Language Models, and the Social Nature of Scientific Success](https://mlsanity.com/paper/2503.20088/) (2025)
- [Translationese as a Rational Response to Translation Task Difficulty](https://mlsanity.com/paper/2603.12050/) (2026)
- [Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models](https://mlsanity.com/paper/2607.16549/) (2026)

## Abstract (arXiv)

By positing a relationship between naturalistic reading times and information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008) provides a natural interface between language models and psycholinguistic models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018) that a language model's ability to model reading times is a linear function of its perplexity. By extending Goodkind and Bicknell's analysis to modern neural architectures, we show that the proposed relation does not always hold for Long Short-Term Memory networks, Transformers, and pre-trained models. We introduce an alternate measure of language modeling performance called predictability norm correlation based on Cloze probabilities measured from human subjects. Our new metric yields a more robust relationship between language model quality and psycholinguistic modeling performance that allows for comparison between models with different training configurations.