mlsanityData to 7 Oct 2026

The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech

arXiv 2409.09305 ↗First posted 14 Sep 2024cs.SD cs.AI cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

We present our system (denoted as T05) for the VoiceMOS Challenge (VMC) 2024. Our system was designed for the VMC 2024 Track 1, which focused on the accurate prediction of naturalness mean opinion score (MOS) for high-quality synthetic speech. In addition to a pretrained self-supervised learning (SSL)-based speech feature extractor, our system incorporates a pretrained image feature extractor to capture the difference of synthetic speech observed in speech spectrograms. We first separately train two MOS predictors that use either of an SSL-based or spectrogram-based feature. Then, we fine-tune the two predictors for better MOS prediction using the fusion of two extracted features. In the VMC 2024 Track 1, our T05 system achieved first place in 7 out of 16 evaluation metrics and second place in the remaining 9 metrics, with a significant difference compared to those ranked third and below. We also report the results of our ablation study to investigate essential factors of our system.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

20 of 31 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  9. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  10. 2020Better Aggregation in Test-Time Augmentationcited by 121
  11. 2021Generalization Ability of MOS Prediction Networkscited by 69
  12. 2022The VoiceMOS Challenge 2022cited by 57
8 more
  1. 2021How do Voices from Past Speech Synthesis Challenges Compare Today?cited by 21
  2. 2022SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesiscited by 18
  3. 2024The VoiceMOS Challenge 2024: Beyond Speech Quality Predictioncited by 17
  4. 2023Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speechcited by 13
  5. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  6. 2023The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domainscited by 8
  7. 2023MOSPC: MOS Prediction Based on Pairwise Comparisoncited by 7
  8. 2023LE-SSL-MOS: Self-Supervised Learning MOS Prediction with Listener Enhancementcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 47.

78 papers.

Sort:

By area:

  1. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  2. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  3. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  4. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
  5. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  6. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  7. 2025SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understandingcited by 6
  8. 2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 5
  9. 2025Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinesecited by 4
  10. 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 4
  11. 2026GSRM: Generative Speech Reward Model for Speech RLHFcited by 4
  12. 2026Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarizationcited by 4
38 more
  1. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 4
  2. 2026Raon-Speech Technical Reportcited by 4
  3. 2025Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancementcited by 3
  4. 2026Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesiscited by 3
  5. 2025SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformercited by 2
  6. 2025APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speechcited by 2
  7. 2025Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesiscited by 2
  8. 2025Length-Aware Rotary Position Embedding for Text-Speech Alignmentcited by 2
  9. 2025DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptationcited by 2
  10. 2026Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systemscited by 2
  11. 2026TriageSim: A Conversational Emergency Triage Simulation Framework from Structured Electronic Health Recordscited by 2
  12. 2025SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech Systemcited by 1
  13. 2025SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Developmentcited by 1
  14. 2025Discrete Optimal Transport and Voice Conversioncited by 1
  15. 2025CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloningcited by 1
  16. 2025UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speechcited by 1
  17. 2025MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidancescited by 1
  18. 2025Provable Speech Attributes Conversion via Latent Independencecited by 1
  19. 2026Geneses: Unified Generative Speech Enhancement and Separationcited by 1
  20. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 1
  21. 2026JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructionscited by 1
  22. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1
  23. 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curationcited by 1
  24. 2026OpenBibleTTS: Large-Scale Speech Resources and TTS Models for Low-Resource Languagescited by 1
  25. 2026DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Eventscited by 1
  26. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 1
  27. 2025Score-Based Training for Energy-Based TTS Modelscited by 0
  28. 2025Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speechcited by 0
  29. 2025ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputscited by 0
  30. 2025Discrete optimal transport is a strong audio adversarial attackcited by 0
  31. 2025Discrete-Time Diffusion-Like Models for Speech Synthesiscited by 0
  32. 2025Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generationcited by 0
  33. 2025Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditionscited by 0
  34. 2025TokenChain: A Discrete Speech Chain via Semantic Token Modelingcited by 0
  35. 2025Contract-Driven QoE Auditing for Speech and Singing Services: From MOS Regression to Service Graphscited by 0
  36. 2026Decoding Order Matters in Autoregressive Speech Synthesiscited by 0
  37. 2026PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMscited by 0
  38. 2026Improving X-Codec-2.0 for Multi-Lingual Speech: 25 Hz Latent Rate and 24 kHz Samplingcited by 0

50 most cited of 78 All 78 (JSON)