mlsanityData to 7 Oct 2026

M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis

arXiv 2512.04720 ↗First posted 4 Dec 2025cs.SDcited by 6

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Non-autoregressive (NAR) text-to-speech synthesis relies on length alignment between text sequences and audio representations, constraining naturalness and expressiveness. Existing methods depend on duration modeling or pseudo-alignment strategies that severely limit naturalness and computational efficiency. We propose M3-TTS, a concise and efficient NAR TTS paradigm based on multi-modal diffusion transformer (MM-DiT) architecture. M3-TTS employs joint diffusion transformer layers for cross-modal alignment, achieving stable monotonic alignment between variable-length text-speech sequences without pseudo-alignment requirements. Single diffusion transformer layers further enhance acoustic detail modeling. The framework integrates a mel-vae codec that provides 3* training acceleration. Experimental results on Seed-TTS and AISHELL-3 benchmarks demonstrate that M3-TTS achieves state-of-the-art NAR performance with the lowest word error rates (1.36% English, 1.31% Chinese) while maintaining competitive naturalness scores. Code and demos will be available at https://wwwwxp.github.io/M3-TTS.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 29 of its 30 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

29 of 30 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching16 Jun
  2. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling26 May
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  5. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  6. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep
  7. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding28 Jul

Other references · 22

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  10. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  11. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  12. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
10 more
  1. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  2. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  3. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  4. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  5. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  6. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  7. 2023VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Designcited by 25
  8. 2024VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processingcited by 7
  9. 2025SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codeccited by 5
  10. 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  2. 2026Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesiscited by 3
  3. 2026UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructionscited by 2
  4. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  5. 2026Beyond Two-stage Diffusion TTS: Joint Structure and Content Refinement via Jump Diffusioncited by 0
  6. 2026ZipL-Dialog: Memory-Efficient Long-Form Spoken Dialog Synthesis via Latent Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
  2. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
  3. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
  4. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  5. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  6. 2026ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inference
  7. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
  8. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models