mlsanityData to 14 Oct 2026

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

arXiv 2306.15687 ↗First posted 23 Jun 2023eess.AS cs.CL cs.LGcited by 379

Not on the map. Why?

Cited by 138 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in urlhttps://voicebox.metademolab.com.

Key facts. Cited by 379 papers in the map (data through 14 Oct 2026). Builds on 66 of its 81 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

66 of 81 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022Flow Matching for Generative Modelingcited by 5,702
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
38 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  13. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  14. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  15. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  16. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  17. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  18. 2022High Fidelity Neural Audio Compressioncited by 687
  19. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  20. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  21. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  22. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  23. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  24. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  25. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  26. 2018SDR - half-baked or well done?cited by 423
  27. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  28. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  29. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  30. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  31. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  32. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  33. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  34. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  35. 2020Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Datacited by 201
  36. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  37. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  38. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 29; 2024: 115; 2025: 126; 2026: 109.

379 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  5. 2025Kimi-Audio Technical Reportcited by 222
  6. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  8. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
  9. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  10. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  11. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  12. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 145
38 more
  1. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  2. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  4. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 104
  5. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  6. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  7. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  8. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  9. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  10. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 76
  11. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  12. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  13. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  14. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  15. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  16. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 57
  17. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  18. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  19. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  20. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  21. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  22. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  23. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  24. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  25. 2023Boosting Latent Diffusion with Flow Matchingcited by 44
  26. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  27. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  28. 2023Motion Flow Matching for Human Motion Synthesis and Editingcited by 38
  29. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  30. 2023Bespoke Solvers for Generative Flow Modelscited by 37
  31. 2023Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformerscited by 36
  32. 2025LBM: Latent Bridge Matching for Fast Image-to-Image Translationcited by 36
  33. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  34. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  35. 2023OpenVoice: Versatile Instant Voice Cloningcited by 32
  36. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  37. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 31
  38. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30

50 most cited of 379 All 379 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Audiobox: Unified Audio Generation with Natural Language Prompts
  2. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
  3. 2026Training a Compact, Strong, and Realistic Text-to-Speech Model
  4. 2026VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
  5. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation
  6. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
  7. 2026MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
  8. 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing