# The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech

arXiv: https://arxiv.org/abs/2409.09305 · HTML: https://mlsanity.com/paper/2409.09305/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [The VoiceMOS Challenge 2022](https://mlsanity.com/paper/2203.11389/) (2022)
- [How do Voices from Past Speech Synthesis Challenges Compare Today?](https://mlsanity.com/paper/2105.02373/) (2021)
- [SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.03040/) (2022)
- [The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction](https://mlsanity.com/paper/2409.07001/) (2024)
- [Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech](https://mlsanity.com/paper/2305.10608/) (2023)
- [The Interspeech 2024 Challenge on Speech Processing Using Discrete Units](https://mlsanity.com/paper/2406.07725/) (2024)
- [The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domains](https://mlsanity.com/paper/2310.02640/) (2023)
- [MOSPC: MOS Prediction Based on Pairwise Comparison](https://mlsanity.com/paper/2306.10493/) (2023)
- [LE-SSL-MOS: Self-Supervised Learning MOS Prediction with Listener Enhancement](https://mlsanity.com/paper/2311.10656/) (2023)

## Built on by (most cited first, 30 of 83)

- [STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models](https://mlsanity.com/paper/2507.15375/) (2025)
- [CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching](https://mlsanity.com/paper/2506.00885/) (2025)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://mlsanity.com/paper/2606.11167/) (2026)
- [GenVC: Self-Supervised Zero-Shot Voice Conversion](https://mlsanity.com/paper/2502.04519/) (2025)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [Raon-Speech Technical Report](https://mlsanity.com/paper/2605.23912/) (2026)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding](https://mlsanity.com/paper/2509.14946/) (2025)
- [OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions](https://mlsanity.com/paper/2505.21724/) (2025)
- [From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents](https://mlsanity.com/paper/2605.15104/) (2026)
- [Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese](https://mlsanity.com/paper/2505.11200/) (2025)
- [GSRM: Generative Speech Reward Model for Speech RLHF](https://mlsanity.com/paper/2602.13891/) (2026)
- [Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization](https://mlsanity.com/paper/2604.06138/) (2026)
- [Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement](https://mlsanity.com/paper/2501.13372/) (2025)
- [Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis](https://mlsanity.com/paper/2606.25369/) (2026)
- [SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformer](https://mlsanity.com/paper/2502.11094/) (2025)
- [SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech System](https://mlsanity.com/paper/2503.23108/) (2025)
- [APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech](https://mlsanity.com/paper/2504.20447/) (2025)
- [Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis](https://mlsanity.com/paper/2505.12226/) (2025)
- [Length-Aware Rotary Position Embedding for Text-Speech Alignment](https://mlsanity.com/paper/2509.11084/) (2025)
- [DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptation](https://mlsanity.com/paper/2509.22727/) (2025)
- [Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems](https://mlsanity.com/paper/2602.23266/) (2026)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [TriageSim: A Conversational Emergency Triage Simulation Framework from Structured Electronic Health Records](https://mlsanity.com/paper/2603.10035/) (2026)
- [JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions](https://mlsanity.com/paper/2605.04505/) (2026)
- [JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation](https://mlsanity.com/paper/2606.03168/) (2026)
- [The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech](https://mlsanity.com/paper/2609.13150/) (2026)
- [SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development](https://mlsanity.com/paper/2503.23848/) (2025)
- [Discrete Optimal Transport and Voice Conversion](https://mlsanity.com/paper/2505.04382/) (2025)

## Abstract (arXiv)

We present our system (denoted as T05) for the VoiceMOS Challenge (VMC) 2024. Our system was designed for the VMC 2024 Track 1, which focused on the accurate prediction of naturalness mean opinion score (MOS) for high-quality synthetic speech. In addition to a pretrained self-supervised learning (SSL)-based speech feature extractor, our system incorporates a pretrained image feature extractor to capture the difference of synthetic speech observed in speech spectrograms. We first separately train two MOS predictors that use either of an SSL-based or spectrogram-based feature. Then, we fine-tune the two predictors for better MOS prediction using the fusion of two extracted features. In the VMC 2024 Track 1, our T05 system achieved first place in 7 out of 16 evaluation metrics and second place in the remaining 9 metrics, with a significant difference compared to those ranked third and below. We also report the results of our ablation study to investigate essential factors of our system.