mlsanityData to 7 Oct 2026

MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis

arXiv 2502.18924 ↗First posted 26 Feb 2025eess.AS cs.LG cs.SDcited by 39

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

While recent zero-shot text-to-speech (TTS) models have significantly improved speech quality and expressiveness, mainstream systems still suffer from issues related to speech-text alignment modeling: 1) models without explicit speech-text alignment modeling exhibit less robustness, especially for hard sentences in practical applications; 2) predefined alignment-based models suffer from naturalness constraints of forced alignments. This paper introduces textitMegaTTS 3, a TTS system featuring an innovative sparse alignment algorithm that guides the latent diffusion transformer (DiT). Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space, thereby achieving high naturalness. Moreover, we employ a multi-condition classifier-free guidance strategy for accent intensity adjustment and adopt the piecewise rectified flow technique to accelerate the generation process. Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity. Notably, our system can generate high-quality one-minute speech with only 8 sampling steps. Audio samples are available at https://sditdemo.github.io/sditdemo/.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 51 of its 70 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

51 of 70 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models25 Aug
  3. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models4 Jun
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  5. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  6. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models13 Jun
  7. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  8. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov

Other references · 43

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2016Least Squares Generative Adversarial Networkscited by 1,000
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
31 more
  1. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  3. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  6. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  7. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  8. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  11. 2021A Survey on Neural Speech Synthesiscited by 182
  12. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  13. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  14. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  15. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  16. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  17. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 95
  18. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  19. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 71
  20. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  21. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  22. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41
  23. 2024Towards audio language modeling -- an overviewcited by 40
  24. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  25. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  26. 2024DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidancecited by 11
  27. 2024AccentBox: Towards High-Fidelity Zero-Shot Accent Generationcited by 9
  28. 2023ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representationscited by 4
  29. 2023Accent-VITS:accent transfer for end-to-end TTScited by 3
  30. 2022Accented Text-to-Speech Synthesis with a Conditional Variational Autoencodercited by 2
  31. 2023Multilingual Multiaccented Multispeaker TTS with RADTTScited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 21.

39 papers.

Sort:

By area:

  1. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 22
  2. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Modelscited by 14
  3. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 14
  4. 2025Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentationscited by 11
  5. 2025SlideTailor: Personalized Presentation Slide Generation for Scientific Paperscited by 11
  6. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  7. 2025JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignmentcited by 9
  8. 2026ALIVE: Animate Your World with Lifelike Audio-Video Generationcited by 6
  9. 2025TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesiscited by 5
  10. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  11. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  12. 2025Selective Classifier-free Guidance for Zero-shot Text-to-speechcited by 4
27 more
  1. 2026X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloningcited by 4
  2. 2025JAM-Flow: Joint Audio-Motion Synthesis with Flow Matchingcited by 3
  3. 2025VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editingcited by 3
  4. 2026dots.tts Technical Reportcited by 3
  5. 2026The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMscited by 2
  6. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  7. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matchingcited by 1
  8. 2025DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibrationcited by 1
  9. 2025Eliminating stability hallucinations in llm-based tts models via attention guidancecited by 1
  10. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  11. 2026Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTScited by 1
  12. 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
  13. 2025Attention2Probability: Attention-Driven Terminology Probability Estimation for Robust Speech-to-Text Systemcited by 0
  14. 2025H-LDM: Hierarchical Latent Diffusion Models for Controllable and Interpretable PCG Synthesis from Clinical Metadatacited by 0
  15. 2026DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusioncited by 0
  16. 2026Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Surveycited by 0
  17. 2026CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbingcited by 0
  18. 2026From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interactioncited by 0
  19. 2026RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matchingcited by 0
  20. 2026Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecscited by 0
  21. 2026Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priorscited by 0
  22. 2026EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesiscited by 0
  23. 2026DETECT-3B-Omni is Agnostic of Content and Demographicscited by 0
  24. 2026DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesiscited by 0
  25. 2026Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenizationcited by 0
  26. 2026Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesiscited by 0
  27. 2026Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis
  2. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
  3. 2026ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inference
  4. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  5. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  6. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation