mlsanityData to 14 Oct 2026

CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training

arXiv 2505.17589 ↗First posted 23 May 2025cs.SD cs.AI eess.AScited by 150

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In our prior works, we introduced a scalable streaming speech synthesis model, CosyVoice 2, which integrates a large language model (LLM) and a chunk-aware flow matching (FM) model, and achieves low-latency bi-streaming speech synthesis and human-parity quality. Despite these advancements, CosyVoice 2 exhibits limitations in language coverage, domain diversity, data volume, text formats, and post-training techniques. In this paper, we present CosyVoice 3, an improved model designed for zero-shot multilingual speech synthesis in the wild, surpassing its predecessor in content consistency, speaker similarity, and prosody naturalness. Key features of CosyVoice 3 include: 1) A novel speech tokenizer to improve prosody naturalness, developed via supervised multi-task training, including automatic speech recognition, speech emotion recognition, language identification, audio event detection, and speaker analysis. 2) A new differentiable reward model for post-training applicable not only to CosyVoice 3 but also to other LLM-based speech synthesis models. 3) Dataset Size Scaling: Training data is expanded from ten thousand hours to one million hours, encompassing 9 languages and 18 Chinese dialects across various domains and text formats. 4) Model Size Scaling: Model parameters are increased from 0.5 billion to 1.5 billion, resulting in enhanced performance on our multilingual benchmark due to the larger model capacity. These advancements contribute significantly to the progress of speech synthesis in the wild. We encourage readers to listen to the demo at https://funaudiollm.github.io/cosyvoice3.

Key facts. Cited by 150 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

52 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization3 Apr
  2. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation6 Feb
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  5. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  6. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  7. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching10 Sep
  8. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep

Other references · 44

  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2025Qwen2.5-Omni Technical Reportcited by 736
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
32 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  6. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  7. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  8. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  9. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  10. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  11. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  12. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  13. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  14. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  15. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  16. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  17. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  18. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  19. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  20. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  21. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  22. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  23. 2023An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verificationcited by 42
  24. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 40
  25. 2023MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separationcited by 39
  26. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  27. 2023SECap: Speech Emotion Captioning with Large Language Modelcited by 33
  28. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  29. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  30. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  31. 2024TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touchcited by 10
  32. 2025Unispeaker: A Unified Approach for Multimodality-driven Speaker Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 126.

150 papers.

Sort:

By area:

  1. 2025Qwen3-Omni Technical Reportcited by 497
  2. 2026Qwen3.5-Omni Technical Reportcited by 174
  3. 2026Qwen3-TTS Technical Reportcited by 118
  4. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 27
  5. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  6. 2025Fun-ASR Technical Reportcited by 19
  7. 2025Fun-Audio-Chat Technical Reportcited by 19
  8. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 17
  9. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  10. 2026MOSS-TTS Technical Reportcited by 14
  11. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  12. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 12
38 more
  1. 2025Towards Interactive Intelligence for Digital Humanscited by 10
  2. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Usecited by 9
  3. 2025RoboOmni: Proactive Robot Manipulation in Omni-modal Contextcited by 8
  4. 2026VoxSafeBench: Not Just What Is Said, but Who, How, and Wherecited by 8
  5. 2025Step-Audio-EditX Technical Reportcited by 7
  6. 2026Raon-Speech Technical Reportcited by 7
  7. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  8. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  9. 2026UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Modelscited by 6
  10. 2026IndexTTS 2.5 Technical Reportcited by 6
  11. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generationcited by 6
  12. 2025MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluationcited by 5
  13. 2026ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesiscited by 5
  14. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  15. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 5
  16. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  17. 2025Selective Classifier-free Guidance for Zero-shot Text-to-speechcited by 4
  18. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  19. 2026From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Benchcited by 4
  20. 2026dots.tts Technical Reportcited by 4
  21. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modelingcited by 3
  22. 2025TTS-1 Technical Reportcited by 3
  23. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS systemcited by 3
  24. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scalecited by 3
  25. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuningcited by 3
  26. 2026FlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model with Personalized Voice Cloningcited by 3
  27. 2026Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMscited by 3
  28. 2026FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenescited by 3
  29. 2026SoulX-Singer: Towards High-Quality Zero-Shot Singing Voice Synthesiscited by 3
  30. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  31. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  32. 2026Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTScited by 3
  33. 2026Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speechcited by 3
  34. 2026X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignmentcited by 3
  35. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 3
  36. 2026Multimodal Duplex Interaction Agentcited by 3
  37. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesiscited by 2
  38. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗
  2. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
  3. 2026Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
  4. 2025VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
  5. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
  6. 2023Seamless: Multilingual Expressive and Streaming Speech Translation
  7. 2026Voxtral TTS
  8. 2026Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis