mlsanityData to 14 Oct 2026

CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

arXiv 2506.00885 ↗First posted 1 Jun 2025cs.SD cs.AI eess.AScited by 11

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Generating natural-sounding, multi-speaker dialogue is crucial for applications such as podcast creation, virtual agents, and multimedia content generation. However, existing systems struggle to maintain speaker consistency, model overlapping speech, and synthesize coherent conversations efficiently. In this paper, we introduce CoVoMix2, a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. CoVoMix2 directly predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model, eliminating the reliance on intermediate token representations. To better capture realistic conversational dynamics, we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies. Our approach achieves state-of-the-art performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed. Notably, CoVoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and precise timing control, demonstrating strong generalizability to real-world speech generation scenarios.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 42 of its 52 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

42 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  3. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors17 Jun
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  5. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  6. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching10 Sep
  7. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep

Other references · 35

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  12. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
23 more
  1. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  2. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  3. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  4. 2022Generative Spoken Dialogue Language Modelingcited by 126
  5. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  6. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  7. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 83
  8. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  9. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 69
  10. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  11. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  12. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  13. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  14. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  15. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  16. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  17. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  18. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  19. 2022FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesiscited by 9
  20. 2023M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesiscited by 6
  21. 2025SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generationcited by 6
  22. 2025Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Predictioncited by 5
  23. 2024Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025VibeVoice Technical Reportcited by 29
  2. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  3. 2026JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructionscited by 2
  4. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  5. 2026DeepASMR: LLM-Based Zero-Shot ASMR Speech Generation for Anyone of Any Voicecited by 1
  6. 2026SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?cited by 0
  7. 2026Tutti: Expressive Multi-Singer Synthesis via Structure-Level Timbre Control and Vocal Texture Modelingcited by 0
  8. 2026Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priorscited by 0
  9. 2026ZipL-Dialog: Memory-Efficient Long-Form Spoken Dialog Synthesis via Latent Flow Matchingcited by 0
  10. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  11. 2026From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTScited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue
  2. 2025DialoSpeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching
  3. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗
  4. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  5. 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech
  6. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
  7. 2026Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
  8. 2024DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applications