# Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation

arXiv: https://arxiv.org/abs/2606.31729 · HTML: https://mlsanity.com/paper/2606.31729/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio](https://mlsanity.com/paper/2304.01448/) (2023)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge](https://mlsanity.com/paper/2505.23009/) (2025)
- [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://mlsanity.com/paper/2506.16381/) (2025)
- [TTSDS -- Text-to-Speech Distribution Score](https://mlsanity.com/paper/2407.12707/) (2024)
- [Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration](https://mlsanity.com/paper/2509.19928/) (2025)
- [SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents](https://mlsanity.com/paper/2508.02013/) (2025)
- [SwiftF0: Fast and Accurate Monophonic Pitch Detection](https://mlsanity.com/paper/2508.18440/) (2025)
- [P2VA: Converting Persona Descriptions into Voice Attributes for Fair and Controllable Text-to-Speech](https://mlsanity.com/paper/2505.17093/) (2025)
- [Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play](https://mlsanity.com/paper/2511.01261/) (2025)

## Abstract (arXiv)

Text-to-speech (TTS) evaluation is an open challenge. While the primary target was "naturalness," recent fidelity gains shifted focus toward "appropriateness" and whether speech is correct for its context. In this work, we examine how perception changes when the expected downstream use varies. We measure the appropriateness and human-likeness of five SOTA TTS systems across five domains: AI assistant, reader, actor, animated character, and spontaneous speaker. Results show appropriateness varies across domains independently of naturalness. While systems shine at reading, expressive domains remain challenging, and optimizing for one can degrade others. Furthermore, naturalness scores tend to penalize stylized speech while rewarding spontaneity. Finally, our study also highlights blind spots in one-size-fits-all evaluation metrics across more expressive domains. We demonstrate that TTS performance is not "solved" but depends on the target domain, requiring context-aware evaluation.