# Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese

arXiv: https://arxiv.org/abs/2505.11200 · HTML: https://mlsanity.com/paper/2505.11200/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 17 of 17)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously](https://mlsanity.com/paper/2306.02044/) (2023)
- [StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion](https://mlsanity.com/paper/2409.10058/) (2024)
- [Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation](https://mlsanity.com/paper/2411.12719/) (2024)
- [Speech Quality Assessment in Crowdsourcing: Comparison Category Rating Method](https://mlsanity.com/paper/2104.04371/) (2021)
- [Compact Neural TTS Voices for Accessibility](https://mlsanity.com/paper/2501.17332/) (2025)
- [Disambiguation of Chinese Polyphones in an End-to-End Framework with Semantic Features Extracted by Pre-trained BERT](https://mlsanity.com/paper/2501.01102/) (2025)

## Built on by (most cited first, 4 of 4)

- [SpeechJudge: Towards Human-Level Judgment for Speech Naturalness](https://mlsanity.com/paper/2511.07931/) (2025)
- [Fish Audio S2 Technical Report](https://mlsanity.com/paper/2603.08823/) (2026)
- [TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis](https://mlsanity.com/paper/2604.22225/) (2026)
- [Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction](https://mlsanity.com/paper/2602.24080/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have significantly improved text-to-speech (TTS) systems, enhancing control over speech style, naturalness, and emotional expression, which brings TTS Systems closer to human-level performance. Although the Mean Opinion Score (MOS) remains the standard for TTS System evaluation, it suffers from subjectivity, environmental inconsistencies, and limited interpretability. Existing evaluation datasets also lack a multi-dimensional design, often neglecting factors such as speaking styles, context diversity, and trap utterances, which is particularly evident in Chinese TTS evaluation. To address these challenges, we introduce the Audio Turing Test (ATT), a multi-dimensional Chinese corpus dataset ATT-Corpus paired with a simple, Turing-Test-inspired evaluation protocol. Instead of relying on complex MOS scales or direct model comparisons, ATT asks evaluators to judge whether a voice sounds human. This simplification reduces rating bias and improves evaluation robustness. To further support rapid model development, we also finetune Qwen2-Audio-Instruct with human judgment data as Auto-ATT for automatic evaluation. Experimental results show that ATT effectively differentiates models across specific capability dimensions using its multi-dimensional design. Auto-ATT also demonstrates strong alignment with human evaluations, confirming its value as a fast and reliable assessment tool. The white-box ATT-Corpus and Auto-ATT can be found in ATT Hugging Face Collection (https://huggingface.co/collections/meituan/audio-turing-test-682446320368164faeaf38a4).