mlsanityData to 14 Oct 2026

MoonCast: High-Quality Zero-Shot Podcast Generation

arXiv 2503.14345 ↗First posted 18 Mar 2025eess.AS cs.AI cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize natural podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To generate long audio, we adopt a long-context language model-based audio modeling approach utilizing large-scale long-context speech data. To enhance spontaneity, we utilize a podcast generation module to generate scripts with spontaneous details, which have been empirically shown to be as crucial as the text-to-speech modeling itself. Experiments demonstrate that MoonCast outperforms baselines, with particularly notable improvements in spontaneity and coherence.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

37 of 42 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
25 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  2. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  3. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  4. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  5. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  6. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  7. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  9. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  10. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  12. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  13. 2022Generative Spoken Dialogue Language Modelingcited by 126
  14. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  15. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  16. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  17. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  18. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  19. 2022High Fidelity Speech Enhancement with Band-split RNNcited by 16
  20. 2023AutoPrep: An Automatic Preprocessing Framework for In-the-Wild Speech Datacited by 16
  21. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  22. 2021AdaSpeech 3: Adaptive Text to Speech for Spontaneous Stylecited by 11
  23. 2024Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Modelscited by 8
  24. 2023SponTTS: modeling and transferring spontaneous style for TTScited by 7
  25. 2025SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2025VibeVoice Technical Reportcited by 29
  3. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  4. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  5. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  6. 2025VoiceBridge: General Speech Restoration with One-step Latent Bridge Modelscited by 4
  7. 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
  8. 2025TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarioscited by 2
  9. 2025Scaling Spoken Language Models with Syllabic Speech Tokenizationcited by 2
  10. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  11. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  12. 2026PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generationcited by 1
6 more
  1. 2026Context-Aware Multimodal Claim Verification in Spoken Dialoguescited by 1
  2. 2026Bagpiper-TTS: Natural Language Guided Universal Speech Synthesiscited by 1
  3. 2026SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?cited by 0
  4. 2026Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priorscited by 0
  5. 2026On Improving Faithfulness of Podcasts from Documentscited by 0
  6. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0