mlsanityData to 14 Oct 2026

Style Mixture of Experts for Expressive Text-To-Speech Synthesis

arXiv 2406.03637 ↗First posted 5 Jun 2024eess.AS cs.CL cs.LGcited by 6

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recent advances in style transfer text-to-speech (TTS) have improved the expressiveness of synthesized speech. However, encoding stylistic information (e.g., timbre, emotion, and prosody) from diverse and unseen reference speech remains a challenge. This paper introduces StyleMoE, an approach that addresses the issue of learning averaged style representations in the style encoder by creating style experts that learn from subsets of data. The proposed method replaces the style encoder in a TTS framework with a Mixture of Experts (MoE) layer. The style experts specialize by learning from subsets of reference speech routed to them by the gating network, enabling them to handle different aspects of the style space. As a result, StyleMoE improves the style coverage of the style encoder for style transfer TTS. Our experiments, both objective and subjective, demonstrate improved style transfer for diverse and unseen reference speech. The proposed method enhances the performance of existing state-of-the-art style transfer TTS models and represents the first study of style MoE in TTS.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 36 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

23 of 36 references.

References · 23

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  7. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  8. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  10. 2021A Survey on Neural Speech Synthesiscited by 184
  11. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  12. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
11 more
  1. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
  2. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  3. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  4. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 33
  5. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  6. 2018Expressive Speech Synthesis via Modeling Expressions with Variational Autoencodercited by 22
  7. 2021AdaSpeech 3: Adaptive Text to Speech for Spontaneous Stylecited by 11
  8. 2023SponTTS: modeling and transferring spontaneous style for TTScited by 7
  9. 2023StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesiscited by 4
  10. 2022Disentangling Style and Speaker Attributes for TTS Style Transfercited by 3
  11. 2023Ensemble prosody prediction for expressive speech synthesiscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025dots.llm1 Technical Reportcited by 5
  2. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  3. 2025Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulationcited by 3
  4. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2
  5. 2025Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Modelscited by 0
  6. 2026AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
  2. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  3. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
  4. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
  5. 2025ReverBERT: A State Space Model for Efficient Text-Driven Speech Style Transfer
  6. 2025HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesis
  7. 2025AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesis
  8. 2023Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any Voice Conversion using Only Speech Data