mlsanityData to 14 Oct 2026

TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems

arXiv 2506.19441 ↗First posted 24 Jun 2025cs.SD cs.CL eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

Evaluation of Text to Speech (TTS) systems is challenging and resource-intensive. Subjective metrics such as Mean Opinion Score (MOS) are not easily comparable between works. Objective metrics are frequently used, but rarely validated against subjective ones. Both kinds of metrics are challenged by recent TTS systems capable of producing synthetic speech indistinguishable from real speech. In this work, we introduce Text to Speech Distribution Score 2 (TTSDS2), a more robust and improved version of TTSDS. Across a range of domains and languages, it is the only one out of 16 compared metrics to correlate with a Spearman correlation above 0.50 for every domain and subjective score evaluated. We also release a range of resources for evaluating synthetic speech close to real speech: A dataset with over 11,000 subjective opinion score ratings; a pipeline for continually recreating a multilingual test dataset to avoid data leakage; and a continually updated benchmark for TTS in 14 languages.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 51 of its 76 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

51 of 76 references.

References · 51

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
38 more
  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  2. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  3. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  4. 2018Learning Word Vectors for 157 Languagescited by 280
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  6. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  7. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  8. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  9. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
  10. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  12. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
  13. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  14. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
  15. 2022Hybrid Transformers for Music Source Separationcited by 134
  16. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  17. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  18. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 97
  19. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 83
  20. 2020Universal Phone Recognition with a Multilingual Allophone Systemcited by 81
  21. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  22. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  23. 2023Better speech synthesis through scalingcited by 63
  24. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
  25. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  26. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 48
  27. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  28. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 35
  29. 2023OpenVoice: Versatile Instant Voice Cloningcited by 32
  30. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
  31. 2023My Science Tutor (MyST) -- A Large Corpus of Children's Conversational Speechcited by 28
  32. 2023Amphion: An Open-Source Audio, Music and Speech Generation Toolkitcited by 27
  33. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  34. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  35. 2022SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesiscited by 18
  36. 2024The VoiceMOS Challenge 2024: Beyond Speech Quality Predictioncited by 17
  37. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  38. 2024Pheme: Efficient and Conversational Speech Generationcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 2
  2. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  3. 2026Imitation Learning for Elder-Facing Speech Synthesiscited by 0
  4. 2026ZONOS2 Technical Reportcited by 0
  5. 2026Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbingcited by 0
  6. 2026Distributional Metrics for Evaluating Spoken Conversational Systemscited by 0
  7. 2026TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patternscited by 0
  8. 2026A Comprehensive Objective Evaluation of Modern Text-to-Speech for Turkish Using Speech Quality Assessment Modelscited by 0