# GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis

arXiv: https://arxiv.org/abs/2106.15153 · HTML: https://mlsanity.com/paper/2106.15153/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [MultiSpeech: Multi-Speaker Text to Speech with Transformer](https://mlsanity.com/paper/2006.04664/) (2020)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/2007.15256/) (2020)
- [Adversarially Trained End-to-end Korean Singing Voice Synthesis System](https://mlsanity.com/paper/1908.01919/) (2019)
- [BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization](https://mlsanity.com/paper/2002.01953/) (2020)
- [AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN](https://mlsanity.com/paper/2005.05642/) (2020)

## Built on by (most cited first, 18 of 18)

- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)
- [DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs](https://mlsanity.com/paper/2201.11972/) (2022)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [DiffVoice: Text-to-Speech with Latent Diffusion](https://mlsanity.com/paper/2304.11750/) (2023)
- [Deep Performer: Score-to-Audio Music Performance Synthesis](https://mlsanity.com/paper/2202.06034/) (2022)
- [VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement](https://mlsanity.com/paper/2211.10666/) (2022)
- [A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS](https://mlsanity.com/paper/2209.10887/) (2022)
- [DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability](https://mlsanity.com/paper/2406.19135/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Multi-modal Adversarial Training for Zero-Shot Voice Cloning](https://mlsanity.com/paper/2408.15916/) (2024)
- [Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalities](https://mlsanity.com/paper/2502.06803/) (2025)
- [Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/2209.12549/) (2022)
- [FastFit: Towards Real-Time Iterative Neural Vocoder by Replacing U-Net Encoder With Multiple STFTs](https://mlsanity.com/paper/2305.10823/) (2023)
- [Adversarial Training of Denoising Diffusion Model Using Dual Discriminators for High-Fidelity Multi-Speaker TTS](https://mlsanity.com/paper/2308.01573/) (2023)
- [Multi-GradSpeech: Towards Diffusion-based Multi-Speaker Text-to-speech Using Consistent Diffusion Models](https://mlsanity.com/paper/2308.10428/) (2023)
- [MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech](https://mlsanity.com/paper/2410.03192/) (2024)

## Abstract (arXiv)

Recent advances in neural multi-speaker text-to-speech (TTS) models have enabled the generation of reasonably good speech quality with a single model and made it possible to synthesize the speech of a speaker with limited training data. Fine-tuning to the target speaker data with the multi-speaker model can achieve better quality, however, there still exists a gap compared to the real speech sample and the model depends on the speaker. In this work, we propose GANSpeech, which is a high-fidelity multi-speaker TTS model that adopts the adversarial training method to a non-autoregressive multi-speaker TTS model. In addition, we propose simple but efficient automatic scaling methods for feature matching loss used in adversarial training. In the subjective listening tests, GANSpeech significantly outperformed the baseline multi-speaker FastSpeech and FastSpeech2 models, and showed a better MOS score than the speaker-specific fine-tuned FastSpeech2.