# CycleSpeech: Reciprocal Alignment for Instruction-Controlled Speech Synthesis and Paralinguistic Understanding

arXiv: https://arxiv.org/abs/2609.24771 · HTML: https://mlsanity.com/paper/2609.24771/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models.
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Placed in

- Speech and audio language models / [Audio language models and spoken dialogue](https://mlsanity.com/speech/audio-language-models-and-spoken-dialogue/)

## Builds on (most cited first, 30 of 32)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [MiMo-Audio: Audio Language Models are Few-Shot Learners](https://mlsanity.com/paper/2512.23808/) (2025)
- [TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models](https://mlsanity.com/paper/2308.14430/) (2023)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement](https://mlsanity.com/paper/2502.07243/) (2025)
- [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017)
- [VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling](https://mlsanity.com/paper/2408.15676/) (2024)
- [Cycle-consistency training for end-to-end speech recognition](https://mlsanity.com/paper/1811.01690/) (2018)
- [SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description](https://mlsanity.com/paper/2408.13608/) (2024)
- [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://mlsanity.com/paper/2506.16381/) (2025)
- [Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency](https://mlsanity.com/paper/1910.11958/) (2019)
- [Audio-Aware Large Language Models as Judges for Speaking Styles](https://mlsanity.com/paper/2506.05984/) (2025)
- [FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System](https://mlsanity.com/paper/2603.10420/) (2026)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructions](https://mlsanity.com/paper/2601.04656/) (2026)
- [Fish Audio S2 Technical Report](https://mlsanity.com/paper/2603.08823/) (2026)
- [OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech](https://mlsanity.com/paper/2601.01459/) (2026)
- [StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models](https://mlsanity.com/paper/2311.16509/) (2023)
- [Speaker consistency loss and step-wise optimization for semi-supervised joint training of TTS and ASR using unpaired text data](https://mlsanity.com/paper/2207.04659/) (2022)
- [ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speech](https://mlsanity.com/paper/2601.12289/) (2026)
- [ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining](https://mlsanity.com/paper/2603.28737/) (2026)

## Abstract (arXiv)

Instruction-controlled speech synthesis and paralinguistic understanding are often trained independently, leaving reciprocal feedback between the two tasks underexplored. We introduce CycleSpeech, a framework that connects generation and understanding through a shared, structured voice profile that serves as a common target for supervision and reciprocal feedback. The forward cycle assesses whether synthesized speech expresses the intended attributes by comparing recovered and target profiles. The backward cycle evaluates whether profiles inferred from real speech can guide reconstruction of the source speaking style. To support both directions, we construct a bilingual dataset of 20,046 examples pairing instructions, target speech, speaker references, and structured profiles. Building on joint supervised fine-tuning, CycleGRPO alternates policy updates using reciprocal rewards grounded in profile consistency and speaking-style reconstruction. Fixed target profiles anchor feedback from the evolving counterpart. This procedure requires neither human preference annotations nor an additional preference-trained reward model. Evaluations on Chinese and English benchmarks show improved instruction adherence and profile recovery while maintaining competitive synthesis quality. Compared with Step-Audio-2-mini, CycleSpeech improves instruction-match accuracy by 4.50 and 10.06 percentage points in Chinese and English, respectively. Controlled ablations further support the contribution of cycle feedback to generation control. These results support structured voice profiles as an interface for reciprocal training between speech generation and paralinguistic understanding. An online demo is available at https://cyclespeech.github.io.