mlsanityData to 7 Oct 2026

VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

arXiv 2403.16973 ↗First posted 25 Mar 2024eess.AS cs.AI cs.CLcited by 118

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALLE and the popular commercial model XTTS-v2. Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named RealEdit. We encourage readers to listen to the demos at https://jasonppy.github.io/VoiceCraft_web.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

46 of 66 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  12. 2023Simple and Controllable Music Generationcited by 480
34 more
  1. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  4. 2022AudioGen: Textually Guided Audio Generationcited by 292
  5. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  6. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  7. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  8. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  9. 2021ASVspoof 2021: accelerating progress in spoofed and deepfake speech detectioncited by 157
  10. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  11. 2020Enabling Language Models to Fill in the Blankscited by 121
  12. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  13. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  14. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  15. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  16. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  17. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  18. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  19. 2020One-class Learning Towards Synthetic Voice Spoofing Detectioncited by 59
  20. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  21. 2023SingSong: Generating musical accompaniments from singingcited by 54
  22. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  23. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  24. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  25. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  26. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41
  27. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  28. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  29. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 33
  30. 2021EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusioncited by 31
  31. 2023FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Modelscited by 23
  32. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  33. 2021Context-Aware Prosody Correction for Text-Based Speech Editingcited by 13
  34. 2022RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 59; 2026: 26.

118 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  3. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  4. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  6. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
  7. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  8. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  9. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  10. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  11. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  12. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
38 more
  1. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  2. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  3. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 22
  4. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  5. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  6. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 17
  7. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 15
  8. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 15
  9. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  10. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  11. 2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 12
  12. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  13. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  14. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  15. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  16. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  17. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  18. 2024Estimating the Completeness of Discrete Speech Unitscited by 9
  19. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  20. 2024LLMs in Education: Novel Perspectives, Challenges, and Opportunitiescited by 7
  21. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  22. 2025UITron-Speech: Towards Automated GUI Agents Based on Speech Instructionscited by 7
  23. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  24. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
  25. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 6
  26. 2025Lotus: Creating Short Videos From Long Videos With Abstractive and Extractive Summarizationcited by 6
  27. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6
  28. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  29. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  30. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5
  31. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Samplingcited by 5
  32. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  33. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  34. 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
  35. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  36. 2026MMAE: A Massive Multitask Audio Editing Benchmarkcited by 5
  37. 2025Voice Cloning: Comprehensive Surveycited by 4
  38. 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matchingcited by 4

50 most cited of 118 All 118 (JSON)