mlsanityData to 19 Oct 2026

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

arXiv 2604.01247 ↗First posted 31 Mar 2026cs.SD eess.AScited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 23 of its 30 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

23 of 30 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation14 Sep
  3. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors17 Jun
  4. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model7 Jun
  5. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training31 Jul
  6. 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training18 May
  7. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speech30 Sep
  8. 2021A Survey on Neural Speech Synthesis29 Jun
  9. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,923
  2. 2022Scalable Diffusion Models with Transformerscited by 6,161
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,155
  4. 2018Self-Attention with Relative Position Representationscited by 1,149
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  6. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 315
  7. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 270
  8. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 225
  9. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 179
  10. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 118
  11. 2024VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmarkcited by 41
  12. 2021Simple Attention Module based Speaker Verification with Iterative noisy label detectioncited by 35
2 more
  1. 2024AASIST3: KAN-Enhanced AASIST Speech Deepfake Detection using SSL Features and Additional Regularization for the ASVspoof 2024 Challengecited by 15
  2. 2025GigaAM: Efficient Self-Supervised Learner for Speech Recognitioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs
  2. 2023DPP-TTS: Diversifying prosodic features of speech via determinantal point processes
  3. 2026ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
  4. 2021Learning De-identified Representations of Prosody from Raw Audio
  5. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  6. 2024DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles
  7. 2021Improving multi-speaker TTS prosody variance with a residual encoder and normalizing flows
  8. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS