# The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

arXiv: https://arxiv.org/abs/2409.07001 · HTML: https://mlsanity.com/paper/2409.07001/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2024.

## Built on by (most cited first, 17 of 17)

- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Aligning Text-to-Music Evaluation with Human Preferences](https://mlsanity.com/paper/2503.16669/) (2025)
- [SpeechJudge: Towards Human-Level Judgment for Speech Naturalness](https://mlsanity.com/paper/2511.07931/) (2025)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment](https://mlsanity.com/paper/2510.01812/) (2025)
- [Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement](https://mlsanity.com/paper/2501.13372/) (2025)
- [Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens](https://mlsanity.com/paper/2509.20485/) (2025)
- [RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems](https://mlsanity.com/paper/2607.14846/) (2026)
- [Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings](https://mlsanity.com/paper/2509.03292/) (2025)
- [Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions](https://mlsanity.com/paper/2509.24457/) (2025)
- [Contract-Driven QoE Auditing for Speech and Singing Services: From MOS Regression to Service Graphs](https://mlsanity.com/paper/2512.04827/) (2025)
- [UR-BERT: Scaling Text Encoders for Massively Multilingual TTS Through Universal Romanization and Speech Token Prediction](https://mlsanity.com/paper/2606.11681/) (2026)
- [PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets](https://mlsanity.com/paper/2606.19597/) (2026)
- [Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations](https://mlsanity.com/paper/2606.19951/) (2026)
- [Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions](https://mlsanity.com/paper/2608.09930/) (2026)
- [DAMOS: Learning Distortion-Aware Speech Quality Assessment through Explicit Distortion Localization](https://mlsanity.com/paper/2608.21176/) (2026)

## Abstract (arXiv)

We present the third edition of the VoiceMOS Challenge, a scientific initiative designed to advance research into automatic prediction of human speech ratings. There were three tracks. The first track was on predicting the quality of ``zoomed-in'' high-quality samples from speech synthesis systems. The second track was to predict ratings of samples from singing voice synthesis and voice conversion with a large variety of systems, listeners, and languages. The third track was semi-supervised quality prediction for noisy, clean, and enhanced speech, where a very small amount of labeled training data was provided. Among the eight teams from both academia and industry, we found that many were able to outperform the baseline systems. Successful techniques included retrieval-based methods and the use of non-self-supervised representations like spectrograms and pitch histograms. These results showed that the challenge has advanced the field of subjective speech rating prediction.