# Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?

arXiv: https://arxiv.org/abs/2311.09325 · HTML: https://mlsanity.com/paper/2311.09325/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks](https://mlsanity.com/paper/2003.06820/) (2020)
- [Human Sentence Processing: Recurrence or Attention?](https://mlsanity.com/paper/2005.09471/) (2020)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://mlsanity.com/paper/2304.11389/) (2023)
- [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022)
- [Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?](https://mlsanity.com/paper/2310.17774/) (2023)
- [Can Entropy Explain Successor Surprisal Effects in Reading?](https://mlsanity.com/paper/1810.11481/) (2018)

## Built on by (most cited first, 7 of 7)

- [Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection](https://mlsanity.com/paper/2403.13335/) (2024)
- [The New Agronomists: Language Models are Experts in Crop Management](https://mlsanity.com/paper/2403.19839/) (2024)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Models](https://mlsanity.com/paper/2406.10882/) (2024)
- [Towards culturally-appropriate conversational AI for health in the majority world: An exploratory study with citizens and professionals in Latin America](https://mlsanity.com/paper/2507.01719/) (2025)
- [Leveraging NTPs for Efficient Hallucination Detection in VLMs](https://mlsanity.com/paper/2509.20379/) (2025)
- [Information Dynamics of Language Communication](https://mlsanity.com/paper/2606.30096/) (2026)

## Abstract (arXiv)

A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word's negative log probability in context. However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. One reason could be that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. Firstly, we show that calibration of large language models typically improves with model size, i.e. poorer calibration cannot account for poorer fit to reading times. Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora. Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens.