# To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times

arXiv: https://arxiv.org/abs/2603.12105 · HTML: https://mlsanity.com/paper/2603.12105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Prompting is not a substitute for probability measurements in large language models](https://mlsanity.com/paper/2305.13264/) (2023)
- [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017)
- [How to Compute the Probability of a Word](https://mlsanity.com/paper/2406.14561/) (2024)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina](https://mlsanity.com/paper/2410.19599/) (2024)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal](https://mlsanity.com/paper/2408.16012/) (2024)
- [Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings](https://mlsanity.com/paper/2509.14405/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have recently been shown to produce estimates of psycholinguistic norms, such as valence, arousal, or concreteness, for words and multiword expressions, that correlate with human judgments. These estimates are obtained by prompting an LLM, in zero-shot fashion, with a question similar to those used in human studies. Meanwhile, for other norms such as lexical decision time or age of acquisition, LLMs require supervised fine-tuning to obtain results that align with ground-truth values. In this paper, we extend this approach to the previously unstudied features of sentence memorability and reading times, which involve the relationship between multiple words in a sentence-level context. Our results show that via fine-tuning, models can provide estimates that correlate with human-derived norms and exceed the predictive power of interpretable baseline predictors, demonstrating that LLMs contain useful information about sentence-level features. At the same time, our results show very mixed zero-shot and few-shot performance, providing further evidence that care is needed when using LLM-prompting as a proxy for human cognitive measures.