# Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

arXiv: https://arxiv.org/abs/2505.16277 · HTML: https://mlsanity.com/paper/2505.16277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 26 of 26)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [Analyzing Cognitive Plausibility of Subword Tokenization](https://mlsanity.com/paper/2310.13348/) (2023)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)
- [[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2404.06214/) (2024)
- [RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2406.19232/) (2024)
- [Is Child-Directed Speech Effective Training Data for Language Models?](https://mlsanity.com/paper/2408.03617/) (2024)
- [On the importance of pre-training data volume for compact language models](https://mlsanity.com/paper/2010.03813/) (2020)
- [CLIMB: Curriculum Learning for Infant-inspired Model Building](https://mlsanity.com/paper/2311.08886/) (2023)
- [Predicting Prosodic Prominence from Text with Pre-trained Contextualized Word Representations](https://mlsanity.com/paper/1908.02262/) (2019)
- [Quantifying the redundancy between prosody and text](https://mlsanity.com/paper/2311.17233/) (2023)
- [What does BERT learn about prosody?](https://mlsanity.com/paper/2304.12706/) (2023)

## Abstract (arXiv)

The achievements of Large Language Models in Natural Language Processing, especially for high-resource languages, call for a better understanding of their characteristics from a cognitive perspective. Researchers have attempted to evaluate artificial models by testing their ability to predict behavioral (e.g., eye-tracking fixations) and physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables (speech reductions, prosodic prominences) and applying them in a similar fashion. More precisely, we extract. We then test models trained with a standard procedure on different pretraining datasets (written, spoken, and mixed genres) for their ability to predict these two variables. Our results show that, after some fine-tuning, the models can predict these production variables well above baselines. We also observe that spoken genre training data provides more accurate predictions than written genres. These results contribute to the broader effort of using high-quality speech corpora as benchmarks for LLMs.