# How do Voices from Past Speech Synthesis Challenges Compare Today?

arXiv: https://arxiv.org/abs/2105.02373 · HTML: https://mlsanity.com/paper/2105.02373/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2021.

## Built on by (most cited first, 21 of 21)

- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech](https://mlsanity.com/paper/2110.09103/) (2021)
- [SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.03040/) (2022)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech](https://mlsanity.com/paper/2402.16321/) (2024)
- [RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier](https://mlsanity.com/paper/2410.22124/) (2024)
- [Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators](https://mlsanity.com/paper/2505.18601/) (2025)
- [From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling](https://mlsanity.com/paper/2510.00743/) (2025)
- [Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features](https://mlsanity.com/paper/2211.00342/) (2022)
- [Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection](https://mlsanity.com/paper/2306.12040/) (2023)
- [TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis](https://mlsanity.com/paper/2604.22225/) (2026)
- [JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions](https://mlsanity.com/paper/2605.04505/) (2026)
- [Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset](https://mlsanity.com/paper/2209.06358/) (2022)
- [To what extent can ASV systems naturally defend against spoofing attacks?](https://mlsanity.com/paper/2406.05339/) (2024)
- [Comparison of Speech Representations for the MOS Prediction System](https://mlsanity.com/paper/2206.13817/) (2022)
- [Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge](https://mlsanity.com/paper/2402.01413/) (2024)
- [Toward Robust Semi-supervised Regression via Dual-stream Knowledge Distillation](https://mlsanity.com/paper/2508.14082/) (2025)
- [Contrastive Learning for Semi-Supervised Deep Regression with Generalized Ordinal Rankings from Spectral Seriation](https://mlsanity.com/paper/2512.09267/) (2025)
- [Augmenting Dysarthric Speech Severity Assessment with MOS Supervision](https://mlsanity.com/paper/2606.18645/) (2026)
- [XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts](https://mlsanity.com/paper/2609.24770/) (2026)

## Abstract (arXiv)

Shared challenges provide a venue for comparing systems trained on common data using a standardized evaluation, and they also provide an invaluable resource for researchers when the data and evaluation results are publicly released. The Blizzard Challenge and Voice Conversion Challenge are two such challenges for text-to-speech synthesis and for speaker conversion, respectively, and their publicly-available system samples and listening test results comprise a historical record of state-of-the-art synthesis methods over the years. In this paper, we revisit these past challenges and conduct a large-scale listening test with samples from many challenges combined. Our aims are to analyze and compare opinions of a large number of systems together, to determine whether and how opinions change over time, and to collect a large-scale dataset of a diverse variety of synthetic samples and their ratings for further research. We found strong correlations challenge by challenge at the system level between the original results and our new listening test. We also observed the importance of the choice of speaker on synthesis quality.