mlsanityData to 7 Oct 2026

CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

arXiv 2407.05407 ↗First posted 7 Jul 2024cs.SD cs.AI eess.AScited by 323

Not on the map. Why?

Cited by 57 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.

Key facts. Cited by 323 papers in the map (data through 7 Oct 2026). Builds on 13 of its 31 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

13 of 31 references.

References · 13

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  6. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  7. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  8. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  9. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  10. 2023Better speech synthesis through scalingcited by 63
  11. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  12. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
1 more
  1. 2023ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speechcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 156; 2026: 136.

323 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  4. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  5. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  9. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  10. 2026Qwen3-TTS Technical Reportcited by 106
  11. 2025Step-Audio 2 Technical Reportcited by 101
  12. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
38 more
  1. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  2. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  3. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  5. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  6. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 76
  7. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  8. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  9. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  10. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  11. 2024Language Model Can Listen While Speakingcited by 62
  12. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  13. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  14. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  15. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  16. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  17. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  18. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  19. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  20. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  21. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  22. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  23. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 27
  24. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  25. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  26. 2025MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audiocited by 24
  27. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
  28. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 22
  29. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  30. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  31. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  32. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  33. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  34. 2025Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensicscited by 18
  35. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  36. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  37. 2025Fun-Audio-Chat Technical Reportcited by 17
  38. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16

50 most cited of 323 All 323 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
  2. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
  3. 2026Voxtral TTS
  4. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  5. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
  6. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition
  7. 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMs
  8. 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech