# How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

arXiv: https://arxiv.org/abs/2609.39032 · HTML: https://mlsanity.com/paper/2609.39032/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 15 of 15)

- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://mlsanity.com/paper/2410.06675/) (2024)
- [Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment](https://mlsanity.com/paper/2502.05356/) (2025)
- [The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement](https://mlsanity.com/paper/2307.03533/) (2023)
- [Lessons Learned from the URGENT 2024 Speech Enhancement Challenge](https://mlsanity.com/paper/2506.01611/) (2025)
- [MOSPC: MOS Prediction Based on Pairwise Comparison](https://mlsanity.com/paper/2306.10493/) (2023)
- [Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment](https://mlsanity.com/paper/2506.12260/) (2025)
- [UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment](https://mlsanity.com/paper/2601.18438/) (2026)
- [Efficient Speech Enhancement via Embeddings from Pre-trained Generative Audioencoders](https://mlsanity.com/paper/2506.11514/) (2025)
- [Resource-Efficient Fine-Tuning Strategies for Automatic MOS Prediction in Text-to-Speech for Low-Resource Languages](https://mlsanity.com/paper/2305.19396/) (2023)
- [Geneses: Unified Generative Speech Enhancement and Separation](https://mlsanity.com/paper/2601.18456/) (2026)
- [URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition](https://mlsanity.com/paper/2506.23874/) (2025)

## Abstract (arXiv)

We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.