# FNH-TTS: Mixture-of-Experts Duration Modeling for Robust Neural Speech Synthesis

arXiv: https://arxiv.org/abs/2508.12001 · HTML: https://mlsanity.com/paper/2508.12001/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 25 of 25)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models](https://mlsanity.com/paper/2308.14430/) (2023)
- [Chunked Autoregressive GAN for Conditional Waveform Synthesis](https://mlsanity.com/paper/2110.10139/) (2021)
- [StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis](https://mlsanity.com/paper/2205.15439/) (2022)
- [VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design](https://mlsanity.com/paper/2307.16430/) (2023)
- [Avocodo: Generative Adversarial Network for Artifact-free Vocoder](https://mlsanity.com/paper/2206.13404/) (2022)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform](https://mlsanity.com/paper/2210.15975/) (2022)
- [Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech](https://mlsanity.com/paper/2406.05401/) (2024)

## Abstract (arXiv)

Natural and human-like speech depends on the coordination between prosodic timing and acoustic realization: duration modeling shapes rhythmic structure, while waveform generation determines whether that structure is rendered naturally. In natural speech, duration patterns vary across linguistic contexts and speakers, requiring a TTS system both to capture this variability and to faithfully realize it in the waveform. To address these challenges, we propose FNH-TTS, a VITS-based end-to-end system that jointly improves duration modeling and waveform generation. A mixture-of-experts duration predictor (MoE-DP) uses multiple experts and routing jointly conditioned on linguistic context and speaker information to model diverse duration patterns. For waveform generation, we adopt an inverse short-time Fourier transform (ISTFT)-based generator, providing a more direct and efficient synthesis path. We further employ multi-resolution and sub-band discriminators for fine-grained temporal and spectral adversarial supervision, thereby supporting natural waveform synthesis. Experiments on LJSpeech, VCTK, and LibriTTS show that FNH-TTS achieves the highest mean MOS on LJSpeech and VCTK and the highest duration-category accuracy on LibriTTS among the compared systems, together with competitive waveform reconstruction and substantially faster vocoder inference. Controlled analyses further show that MoE-DP primarily drives the duration-modeling gains, while the vocoder-side components make complementary contributions to synthesis quality and efficiency.