mlsanityData to 7 Oct 2026

VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing

arXiv 2511.12347 ↗First posted 15 Nov 2025eess.AS cs.CL cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce VoiceCraft-X, an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot Text-to-Speech (TTS) synthesis across 11 languages: English, Mandarin, Korean, Japanese, Spanish, French, German, Dutch, Italian, Portuguese, and Polish. VoiceCraft-X utilizes the Qwen3 large language model for phoneme-free cross-lingual text processing and a novel token reordering mechanism with time-aligned text and speech tokens to handle both tasks as a single sequence generation problem. The model generates high-quality, natural-sounding speech, seamlessly creating new audio or editing existing recordings within one framework. VoiceCraft-X shows robust performance in diverse linguistic settings, even with limited per-language data, underscoring the power of unified autoregressive approaches for advancing complex, real-world multilingual speech applications. Audio samples are available at https://zhishengzheng.com/voicecraft-x/.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

32 of 47 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  8. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  9. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  10. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  11. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  12. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
20 more
  1. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  2. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  3. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  4. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  5. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  6. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  7. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
  8. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
  9. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  10. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  11. 2024WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmarkcited by 39
  12. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  13. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  14. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  15. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  16. 2020DiDiSpeech: A Large Scale Mandarin Speech Corpuscited by 33
  17. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  18. 2022SpeechPainter: Text-conditioned Speech Inpaintingcited by 17
  19. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  20. 2024Extending Multilingual Speech Synthesis to 100+ Languages without Transcribed Datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 22
  2. 2026MMAE: A Massive Multitask Audio Editing Benchmarkcited by 5
  3. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speechcited by 0