# InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

arXiv: https://arxiv.org/abs/2609.36287 · HTML: https://mlsanity.com/paper/2609.36287/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models.
- Builds on 23 of its 24 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Speech and audio language models / [Audio language models and spoken dialogue](https://mlsanity.com/speech/audio-language-models-and-spoken-dialogue/)

## Builds on (most cited first, 23 of 23)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [Zero-shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2411.09943/) (2024)
- [VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling](https://mlsanity.com/paper/2408.15676/) (2024)
- [3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization](https://mlsanity.com/paper/2403.19971/) (2024)
- [VoiceSculptor: Your Voice, Designed By You](https://mlsanity.com/paper/2601.10629/) (2026)
- [OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech](https://mlsanity.com/paper/2601.01459/) (2026)
- [LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models](https://mlsanity.com/paper/2601.04233/) (2026)
- [Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm](https://mlsanity.com/paper/2607.23938/) (2026)
- [MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control](https://mlsanity.com/paper/2604.21164/) (2026)
- [FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations](https://mlsanity.com/paper/2608.17492/) (2026)
- [HD-PPT: Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTS](https://mlsanity.com/paper/2509.19001/) (2025)
- [WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS](https://mlsanity.com/paper/2607.06461/) (2026)
- [How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2606.20532/) (2026)

## Abstract (arXiv)

Instruction-based text-to-speech (ITTS) systems enable natural-language control of expressive speech generation, but often offer limited transparency and fine-grained control over individual text units. Character-level controllable TTS systems provide explicit acoustic control, yet typically rely on user-specified acoustic attributes. To bridge this gap, we propose InstCharVoice, a unified framework that grounds natural-language instructions in character-level acoustic control. We first construct grounded instruction annotations on the WordVoice-5A-zh corpus using Qwen3-Omni. With this supervision, we train an autoregressive model to identify instruction-relevant characters and predict their acoustic attributes before generating the corresponding speech tokens. Keyword prediction and grounding-aware loss weighting help the model focus on instruction-relevant characters and attributes. Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability. Audio samples are available at https://xxh333.github.io/instcharvoice-demo/.