# EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge

arXiv: https://arxiv.org/abs/2505.23009 · HTML: https://mlsanity.com/paper/2505.23009/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 105 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 13 of 13)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [SQuId: Measuring Speech Naturalness in Many Languages](https://mlsanity.com/paper/2210.06324/) (2022)
- [Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model](https://mlsanity.com/paper/2405.09768/) (2024)
- [Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptation](https://mlsanity.com/paper/2504.08274/) (2025)

## Built on by (most cited first, 24 of 24)

- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows](https://mlsanity.com/paper/2512.08769/) (2025)
- [VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions](https://mlsanity.com/paper/2509.09716/) (2025)
- [MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models](https://mlsanity.com/paper/2511.00850/) (2025)
- [Fish Audio S2 Technical Report](https://mlsanity.com/paper/2603.08823/) (2026)
- [Hearing the Order: Investigating Position Bias in Large Audio-Language Models](https://mlsanity.com/paper/2510.00628/) (2025)
- [dots.tts Technical Report](https://mlsanity.com/paper/2606.07080/) (2026)
- [Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS](https://mlsanity.com/paper/2605.30748/) (2026)
- [EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis](https://mlsanity.com/paper/2608.23791/) (2026)
- [SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation](https://mlsanity.com/paper/2604.20842/) (2026)
- [HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding](https://mlsanity.com/paper/2605.29948/) (2026)
- [Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation](https://mlsanity.com/paper/2607.13477/) (2026)
- [RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems](https://mlsanity.com/paper/2607.14846/) (2026)
- [Computational Narrative Understanding for Expressive Text-to-Speech](https://mlsanity.com/paper/2509.04072/) (2025)
- [Scaling Properties of Continuous Diffusion Spoken Language Models](https://mlsanity.com/paper/2604.24416/) (2026)
- [The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation](https://mlsanity.com/paper/2604.26347/) (2026)
- [TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context](https://mlsanity.com/paper/2609.08703/) (2026)
- [JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1](https://mlsanity.com/paper/2507.20987/) (2025)
- [ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge](https://mlsanity.com/paper/2606.24648/) (2026)
- [Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation](https://mlsanity.com/paper/2606.31729/) (2026)
- [LuxEmo: Expressive Text-to-Speech Corpus for Luxembourgish](https://mlsanity.com/paper/2606.31947/) (2026)
- [Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions](https://mlsanity.com/paper/2608.09930/) (2026)
- [Tone on a Budget: A Reference-Free Metric for Lexical Tone in Massively Multilingual Text-to-Speech](https://mlsanity.com/paper/2609.14817/) (2026)
- [Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis](https://mlsanity.com/paper/2609.20124/) (2026)

## Abstract (arXiv)

Text-to-Speech (TTS) benchmarks often fail to capture how well models handle nuanced and semantically complex text. Building on textitEmergentTTS, we introduce textitEmergentTTS-Eval, a comprehensive benchmark covering six challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic complexity, complex pronunciation (e.g. URLs, formulas), and questions. Crucially, our framework automates both test-case generation and evaluation, making the benchmark easily extensible. Starting from a small set of human-written seed prompts, we iteratively extend them using LLMs to target specific structural, phonetic and prosodic challenges, resulting in 1,645 diverse test cases. Moreover, we employ a model-as-a-judge approach, using a Large Audio Language Model (LALM) to assess the speech across multiple dimensions such as expressed emotion, prosodic, intonational, and pronunciation accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems, such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval, demonstrating its ability to reveal fine-grained performance differences. Results show that the model-as-a-judge approach offers robust TTS assessment and a high correlation with human preferences. We open source the evaluation hrefhttps://github.com/boson-ai/EmergentTTS-Eval-publiccode and the hrefhttps://huggingface.co/datasets/bosonai/EmergentTTS-Evaldataset.