mlsanityData to 19 Oct 2026

CycleSpeech: Reciprocal Alignment for Instruction-Controlled Speech Synthesis and Paralinguistic Understanding

arXiv 2609.24771 ↗First posted 21 Sep 2026cs.SDcited by 0

On the map: Speech and audio language models → Audio language models and spoken dialogue

Abstract (arXiv)

Instruction-controlled speech synthesis and paralinguistic understanding are often trained independently, leaving reciprocal feedback between the two tasks underexplored. We introduce CycleSpeech, a framework that connects generation and understanding through a shared, structured voice profile that serves as a common target for supervision and reciprocal feedback. The forward cycle assesses whether synthesized speech expresses the intended attributes by comparing recovered and target profiles. The backward cycle evaluates whether profiles inferred from real speech can guide reconstruction of the source speaking style. To support both directions, we construct a bilingual dataset of 20,046 examples pairing instructions, target speech, speaker references, and structured profiles. Building on joint supervised fine-tuning, CycleGRPO alternates policy updates using reciprocal rewards grounded in profile consistency and speaking-style reconstruction. Fixed target profiles anchor feedback from the evolving counterpart. This procedure requires neither human preference annotations nor an additional preference-trained reward model. Evaluations on Chinese and English benchmarks show improved instruction adherence and profile recovery while maintaining competitive synthesis quality. Compared with Step-Audio-2-mini, CycleSpeech improves instruction-match accuracy by 4.50 and 10.06 percentage points in Chinese and English, respectively. Controlled ablations further support the contribution of cycle feedback to generation control. These results support structured voice profiles as an interface for reciprocal training between speech generation and paralinguistic understanding. An online demo is available at https://cyclespeech.github.io.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

32 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Fish Audio S2 Technical Report9 Mar
  2. 2026OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech4 Jan
  3. 2025MiMo-Audio: Audio Language Models are Few-Shot Learners29 Dec
  4. 2025BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs30 Sep
  5. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems19 Jun
  6. 2024VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling28 Aug

Other references · 26

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  2. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 316
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 215
  5. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 179
  6. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 162
  7. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 157
  8. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 144
  9. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 139
  10. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 101
  11. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 65
  12. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 58
14 more
  1. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 52
  2. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 51
  3. 2018Cycle-consistency training for end-to-end speech recognitioncited by 34
  4. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 33
  5. 2019Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistencycited by 26
  6. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 25
  7. 2026FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition Systemcited by 23
  8. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 17
  9. 2026FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructionscited by 17
  10. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 8
  11. 2022Speaker consistency loss and step-wise optimization for semi-supervised joint training of TTS and ASR using unpaired text datacited by 5
  12. 2026ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speechcited by 3
  13. 2026ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretrainingcited by 2
  14. 2026SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verificationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation
  2. 2026MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
  3. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions
  4. 2026PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis
  5. 2026Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
  6. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis
  7. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS
  8. 2026Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation