mlsanityData to 14 Oct 2026

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

arXiv 2409.00750 ↗First posted 1 Sep 2024cs.SD cs.AI cs.LGcited by 161

Not on the map. Why?

Cited by 46 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Builds on 68 of its 76 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

68 of 76 references.

References · 68

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2022A ConvNet for the 2020scited by 3,579
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
38 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  3. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  9. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  13. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  14. 2023Simple and Controllable Music Generationcited by 488
  15. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  16. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  17. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  18. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  19. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  20. 2022MAGVIT: Masked Generative Video Transformercited by 346
  21. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  22. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 310
  23. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  24. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  25. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  26. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  27. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  28. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  29. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  30. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  31. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  32. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
  33. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  34. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  35. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  36. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  37. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
  38. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 79; 2026: 75.

161 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  4. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  5. 2026Qwen3.5-Omni Technical Reportcited by 174
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  7. 2026Qwen3-TTS Technical Reportcited by 118
  8. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  9. 2025Step-Audio 2 Technical Reportcited by 105
  10. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  11. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  12. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
38 more
  1. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  4. 2025VibeVoice Technical Reportcited by 29
  5. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  6. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  7. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  8. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  9. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  10. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  11. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  12. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  13. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 17
  14. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  15. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  16. 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
  17. 2026MOSS-TTS Technical Reportcited by 14
  18. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  19. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  20. 2024TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touchcited by 10
  21. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  22. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  23. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  24. 2026The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexitycited by 9
  25. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  26. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 8
  27. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
  28. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 7
  29. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  30. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  31. 2025Step-Audio-EditX Technical Reportcited by 7
  32. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  33. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  34. 2025M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesiscited by 6
  35. 2026IndexTTS 2.5 Technical Reportcited by 6
  36. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generationcited by 6
  37. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Samplingcited by 5
  38. 2025Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversioncited by 5

50 most cited of 161 All 161 (JSON)