mlsanityData to 19 Oct 2026

Do You Hear What I Mean? Quantifying the Instruction-Perception Gap in Instruction-Guided Expressive Text-To-Speech Systems

arXiv 2509.13989 ↗First posted 17 Sep 2025eess.AScited by 0

On the map: Speech and audio language models → Audio language models and spoken dialogue

Abstract (arXiv)

Instruction-guided text-to-speech (ITTS) enables users to control speech generation through natural language prompts, offering a more intuitive interface than traditional TTS. However, the alignment between user style instructions and listener perception remains largely unexplored. This work first presents a perceptual analysis of ITTS controllability across two expressive dimensions (adverbs of degree and graded emotion intensity) and collects human ratings on speaker age and word-level emphasis attributes. To comprehensively reveal the instruction-perception gap, we provide a data collection with large-scale human evaluations, named Expressive VOice Control (E-VOC) corpus. Furthermore, we reveal that (1) gpt-4o-mini-tts is the most reliable ITTS model with great alignment between instruction and generated utterances across acoustic dimensions. (2) The 5 analyzed ITTS systems tend to generate Adult voices even when the instructions ask to use child or Elderly voices. (3) Fine-grained control remains a major challenge, indicating that most ITTS systems have substantial room for improvement in interpreting slightly different attribute instructions.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models. Builds on 22 of its 30 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

22 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems19 Jun
  2. 2024VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling28 Aug

Other references · 20

  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,936
  2. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 947
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 380
  4. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 270
  5. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 263
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 215
  7. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 162
  8. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 154
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 144
  10. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 139
  11. 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 123
  12. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 101
8 more
  1. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 58
  2. 2017Word Affect Intensitiescited by 54
  3. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 36
  4. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 33
  5. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 30
  6. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 25
  7. 2025BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insightscited by 6
  8. 2025HighRateMOS: Sampling-Rate Aware Modeling for Speech Quality Assessmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
  2. 2026MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
  3. 2026AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
  4. 2026Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
  5. 2026Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation
  6. 2025InstructAudio: Unified speech and music generation with natural language instruction
  7. 2025BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs
  8. 2022Analysis and Utilization of Entrainment on Acoustic and Emotion Features in User-agent Dialogue