mlsanityData to 14 Oct 2026

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

arXiv 2412.10117 ↗First posted 13 Dec 2024cs.SD cs.AI cs.LGcited by 288

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode. We invite readers to listen to the demos at https://funaudiollm.github.io/cosyvoice2.

Key facts. Cited by 288 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 48 of its 57 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

48 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  3. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  4. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep

Other references · 44

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
32 more
  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  6. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  7. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  8. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  9. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  10. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  12. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  13. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  14. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  15. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  16. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  17. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  18. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  19. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  20. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  21. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  22. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  23. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  24. 2023An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verificationcited by 42
  25. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  26. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  27. 2023OpenVoice: Versatile Instant Voice Cloningcited by 32
  28. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  29. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  30. 2024LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codescited by 13
  31. 2023Speak While You Think: Streaming Speech Synthesis During Text Generationcited by 8
  32. 2024Zero-Shot Text-to-Speech from Continuous Text Streamscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 136; 2026: 152.

288 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2026Qwen3.5-Omni Technical Reportcited by 174
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  5. 2026Qwen3-TTS Technical Reportcited by 118
  6. 2025Step-Audio 2 Technical Reportcited by 105
  7. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  8. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  9. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  10. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  11. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  12. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
38 more
  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  3. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  4. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 31
  5. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  6. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 30
  7. 2025VibeVoice Technical Reportcited by 29
  8. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  9. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  10. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  11. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  12. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  13. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 21
  14. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  15. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  16. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  17. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  18. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  19. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  20. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  21. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizationscited by 15
  22. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 15
  23. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  24. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  25. 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
  26. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 13
  27. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  28. 2025Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systemscited by 13
  29. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  30. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 13
  31. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 11
  32. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  33. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  34. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  35. 2025Towards Interactive Intelligence for Digital Humanscited by 10
  36. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  37. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 9
  38. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 9

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗
  2. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
  3. 2025VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
  4. 2023Seamless: Multilingual Expressive and Streaming Speech Translation
  5. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
  6. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  7. 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models
  8. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers