mlsanityData to 14 Oct 2026

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

arXiv 2608.02235 ↗First posted 3 Aug 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages. However, comprehensive evaluation methodologies that jointly assess perceptual quality, speaker similarity, and acoustic fidelity across diverse speech domains remain limited, particularly for low-resource and underrepresented languages. This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems through domain-specific analysis. The proposed framework integrates complementary subjective and objective evaluation protocols and is demonstrated through a comprehensive case study on a representative low-resource language spanning four speech domains: Formal, Conversational, Literary/Storytelling, and Emotional. Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated using MUSHRA listening tests, ABX discrimination tests, speaker similarity scoring with Resemblyzer, and acoustic analyses based on mel-cepstral distortion (MCD) and F0 RMSE over 960 audio pairs. Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge (mean MCD 12.03 dB; mean F0 RMSE 889 cents), while conversational speech achieves the highest overall acoustic fidelity. Beyond the empirical findings, this work provides a reproducible evaluation framework, publicly releasing evaluation scripts, result tables, and executable Colab notebooks to support standardized benchmarking and future research on TTS evaluation for low-resource languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 26 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

11 of 26 references.

References · 11

  1. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  6. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  7. 2021A Survey on Neural Speech Synthesiscited by 184
  8. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  9. 2022Towards Building Text-To-Speech Systems for the Next Billion Userscited by 27
  10. 2024Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluationcited by 7
  11. 2026PashtoTTS-Bench: automated screening for low-resource non-Latin-script text-to-speechcited by 1

Built on by

Not cited yet.