mlsanityData to 7 Oct 2026

TTSDS -- Text-to-Speech Distribution Score

arXiv 2407.12707 ↗First posted 17 Jul 2024eess.AS cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Many recently published Text-to-Speech (TTS) systems produce audio close to real speech. However, TTS evaluation needs to be revisited to make sense of the results obtained with the new architectures, approaches and datasets. We propose evaluating the quality of synthetic speech as a combination of multiple factors such as prosody, speaker identity, and intelligibility. Our approach assesses how well synthetic speech mirrors real speech by obtaining correlates of each factor and measuring their distance from both real speech datasets and noise datasets. We benchmark 35 TTS systems developed between 2008 and 2024 and show that our score computed as an unweighted average of factors strongly correlates with the human evaluations from each time period.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

29 of 54 references.

References · 29

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018CoQA: A Conversational Question Answering Challengecited by 674
  7. 2022High Fidelity Neural Audio Compressioncited by 673
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  11. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  12. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
17 more
  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  2. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  3. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  4. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  5. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  6. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  7. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
  8. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  9. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  10. 2022The VoiceMOS Challenge 2022cited by 57
  11. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  12. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  13. 2018Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Modelcited by 33
  14. 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
  15. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  16. 2022Evaluating and reducing the distance between synthetic and real speech distributionscited by 8
  17. 2024Pheme: Efficient and Conversational Speech Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  2. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  3. 2025Protect: Towards Robust Guardrailing Stack for Trustworthy Enterprise LLM Systemscited by 3
  4. 2026An Evaluation Framework for Text-to-Speech Voice Reconstructioncited by 1
  5. 2025Text-to-Image Models Leave Identifiable Signatures: Implications for Leaderboard Securitycited by 0
  6. 2026Identifying Models Behind Text-to-Image Leaderboardscited by 0
  7. 2026Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecscited by 0
  8. 2026Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluationcited by 0
  9. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0
  10. 2026Iterative Self-Learning for Expressive Text-to-Speech Synthesiscited by 0
  11. 2026Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speechcited by 0
  12. 2026Continuous-Time Acoustic Modelling with Neural Controlled Differential Equationscited by 0
2 more
  1. 2026Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbingcited by 0
  2. 2026Distributional Metrics for Evaluating Spoken Conversational Systemscited by 0