# Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations

arXiv: https://arxiv.org/abs/2606.19951 · HTML: https://mlsanity.com/paper/2606.19951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis](https://mlsanity.com/paper/2211.09407/) (2022)
- [VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music](https://mlsanity.com/paper/2412.17667/) (2024)
- [ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech](https://mlsanity.com/paper/2409.15897/) (2024)
- [The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction](https://mlsanity.com/paper/2409.07001/) (2024)
- [MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models](https://mlsanity.com/paper/2411.03715/) (2024)
- [Source-Filter HiFi-GAN: Fast and Pitch Controllable High-Fidelity Neural Vocoder](https://mlsanity.com/paper/2210.15533/) (2022)
- [SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Prediction](https://mlsanity.com/paper/2406.10911/) (2024)
- [Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration](https://mlsanity.com/paper/2509.19928/) (2025)
- [SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit](https://mlsanity.com/paper/2505.15061/) (2025)
- [Pitch-and-Spectrum-Aware Singing Quality Assessment with Bias Correction and Model Fusion](https://mlsanity.com/paper/2411.11123/) (2024)
- [Uni-VERSA: Versatile Speech Assessment with a Unified Network](https://mlsanity.com/paper/2505.20741/) (2025)
- [Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features](https://mlsanity.com/paper/2211.00342/) (2022)
- [Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Model](https://mlsanity.com/paper/2412.03074/) (2024)

## Abstract (arXiv)

Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.