FNH-TTS: Mixture-of-Experts Duration Modeling for Robust Neural Speech Synthesis
On the map: Mixture-of-experts → Multimodal MoE models
Abstract (arXiv)
Natural and human-like speech depends on the coordination between prosodic timing and acoustic realization: duration modeling shapes rhythmic structure, while waveform generation determines whether that structure is rendered naturally. In natural speech, duration patterns vary across linguistic contexts and speakers, requiring a TTS system both to capture this variability and to faithfully realize it in the waveform. To address these challenges, we propose FNH-TTS, a VITS-based end-to-end system that jointly improves duration modeling and waveform generation. A mixture-of-experts duration predictor (MoE-DP) uses multiple experts and routing jointly conditioned on linguistic context and speaker information to model diverse duration patterns. For waveform generation, we adopt an inverse short-time Fourier transform (ISTFT)-based generator, providing a more direct and efficient synthesis path. We further employ multi-resolution and sub-band discriminators for fine-grained temporal and spectral adversarial supervision, thereby supporting natural waveform synthesis. Experiments on LJSpeech, VCTK, and LibriTTS show that FNH-TTS achieves the highest mean MOS on LJSpeech and VCTK and the highest duration-category accuracy on LibriTTS among the compared systems, together with competitive waveform reconstruction and substantially faster vocoder inference. Controlled analyses further show that MoE-DP primarily drives the duration-modeling gains, while the vocoder-side components make complementary contributions to synthesis quality and efficiency.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 36 references that are in the map. First posted on arXiv 16 Aug 2025.
Builds on
25 of 36 references.
References · 25
- 2022A ConvNet for the 2020scited by 3,579
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
- 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
- 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
- 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
- 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
13 more
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
- 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
- 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
- 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
- 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
- 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
- 2021Chunked Autoregressive GAN for Conditional Waveform Synthesiscited by 38
- 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 33
- 2023VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Designcited by 25
- 2022Avocodo: Generative Adversarial Network for Artifact-free Vocodercited by 18
- 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
- 2022Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transformcited by 9
- 2024Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speechcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
- 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
- 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis
- 2021On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis
- 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
- 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
- 2026Wave-Trainer-Fit: Neural Vocoder with Trainable Prior and Fixed-Point Iteration towards High-Quality Speech Generation from SSL features ↗
- 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models