# AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

arXiv: https://arxiv.org/abs/2605.17583 · HTML: https://mlsanity.com/paper/2605.17583/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 20 of 20)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System](https://mlsanity.com/paper/2502.05512/) (2025)
- [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://mlsanity.com/paper/2506.16381/) (2025)
- [AlphaAgents: Large Language Model based Multi-Agents for Equity Portfolio Constructions](https://mlsanity.com/paper/2508.11152/) (2025)
- [The MSP-Podcast Corpus](https://mlsanity.com/paper/2509.09791/) (2025)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2508.11256/) (2025)
- [SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems](https://mlsanity.com/paper/2401.03945/) (2024)
- [IndexTTS 2.5 Technical Report](https://mlsanity.com/paper/2601.03888/) (2026)
- [UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts](https://mlsanity.com/paper/2404.18398/) (2024)

## Built on by (most cited first, 1 of 1)

- [PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval](https://mlsanity.com/paper/2609.36923/) (2026)

## Abstract (arXiv)

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement agent mitigates speaker-emotion leakage by learning separable identity and emotion-prosody subspaces with leakage-suppressing regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast-Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic-acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method.