# Language-agnostic, automated assessment of listeners' speech recall using large language models

arXiv: https://arxiv.org/abs/2503.01045 · HTML: https://mlsanity.com/paper/2503.01045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 129 references that are in the map.
- First posted on arXiv 2 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Universal Sentence Representation Learning with Conditional Masked Language Model](https://mlsanity.com/paper/2012.14388/) (2020)
- [Correlation Coefficients and Semantic Textual Similarity](https://mlsanity.com/paper/1905.07790/) (2019)
- [Length-Induced Embedding Collapse in PLM-based Models](https://mlsanity.com/paper/2410.24200/) (2024)
- [Large-scale study of human memory for meaningful narratives](https://mlsanity.com/paper/2311.04742/) (2023)
- [Event Segmentation Applications in Large Language Model Enabled Automated Recall Assessments](https://mlsanity.com/paper/2502.13349/) (2025)

## Abstract (arXiv)

Speech-comprehension difficulties are common among older people. Standard speech tests do not fully capture such difficulties because the tests poorly resemble the context-rich, story-like nature of ongoing conversation and are typically available only in a country's dominant/official language (e.g., English), leading to inaccurate scores for native speakers of other languages. Assessments for naturalistic, story speech in multiple languages require accurate, time-efficient scoring. The current research leverages modern large language models (LLMs) in native English speakers and native speakers of 10 other languages to automate the generation of high-quality, spoken stories and scoring of speech recall in different languages. Participants listened to and freely recalled short stories (in quiet/clear and in babble noise) in their native language. LLM text-embeddings and LLM prompt engineering with semantic similarity analyses to score speech recall revealed sensitivity to known effects of temporal order, primacy/recency, and background noise, and high similarity of recall scores across languages. The work overcomes limitations associated with simple speech materials and testing of closed native-speaker groups because recall data of varying length and details can be mapped across languages with high accuracy. The full automation of speech generation and recall scoring provides an important step towards comprehension assessments of naturalistic speech with clinical applicability.