# The State Of TTS: A Case Study with Human Fooling Rates

arXiv: https://arxiv.org/abs/2508.04179 · HTML: https://mlsanity.com/paper/2508.04179/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 15 of 15)

- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [Does GPT-4 pass the Turing test?](https://mlsanity.com/paper/2310.20216/) (2023)
- [Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech](https://mlsanity.com/paper/2305.10608/) (2023)
- [Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously](https://mlsanity.com/paper/2306.02044/) (2023)
- [Evaluating Long-form Text-to-Speech: Comparing the Ratings of Sentences and Paragraphs](https://mlsanity.com/paper/1909.03965/) (2019)
- [Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation](https://mlsanity.com/paper/2411.12719/) (2024)

## Built on by (most cited first, 4 of 4)

- [Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages](https://mlsanity.com/paper/2604.21481/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [How to Leverage Synthetic Speech for LLM-Based ASR Systems?](https://mlsanity.com/paper/2606.29031/) (2026)
- [When Synthetic Speech Is All You Have: Better Call GRPO](https://mlsanity.com/paper/2607.08409/) (2026)

## Abstract (arXiv)

While subjective evaluations in recent years indicate rapid progress in TTS, can current TTS systems truly pass a human deception test in a Turing-like evaluation? We introduce Human Fooling Rate (HFR), a metric that directly measures how often machine-generated speech is mistaken for human. Our large-scale evaluation of open-source and commercial TTS models reveals critical insights: (i) CMOS-based claims of human parity often fail under deception testing, (ii) TTS progress should be benchmarked on datasets where human speech achieves high HFRs, as evaluating against monotonous or less expressive reference samples sets a low bar, (iii) Commercial models approach human deception in zero-shot settings, while open-source systems still struggle with natural conversational speech; (iv) Fine-tuning on high-quality data improves realism but does not fully bridge the gap. Our findings underscore the need for more realistic, human-centric evaluations alongside existing subjective tests.