Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models
Not on the map. Why?
Abstract (arXiv)
Recent advances in Speech Foundation Models (SFMs) enable direct processing of raw audio, allowing models to respond to subtle paralinguistic variation. However, how these models interpret non-lexical cues remains largely unstudied. We introduce VQ-Bench, a controlled evaluation suite featuring a parallel dataset of synthesized modal, breathy, creaky, and end-creak phonation types. We evaluate SFM sensitivity through open-ended generation across four ecologically valid domains, alongside speech emotion recognition. Our results reveal performance gaps: while a leading commercial API failed basic biometric sanity checks, other models exhibited systematic shifts in agency, empathy, and leadership based on phonation. Our findings also highlight gender asymmetries in salary and leadership endorsements, demonstrating that SFMs may mirror or amplify human social biases. This work establishes a reproducible framework for ensuring responsible paralinguistic interpretation in speech-based AI.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 41 references that are in the map. First posted on arXiv 29 Oct 2025.
Builds on
15 of 41 references.
References · 15
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
- 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
- 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
- 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
- 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
- 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
- 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
- 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
- 2025Do Bias Benchmarks Generalise? Evidence from Voice-based Evaluation of Gender Bias in SpeechLLMscited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.