# Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis

arXiv: https://arxiv.org/abs/2106.10831 · HTML: https://mlsanity.com/paper/2106.10831/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2021.

## Built on by (most cited first, 10 of 10)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models](https://mlsanity.com/paper/2404.00569/) (2024)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE](https://mlsanity.com/paper/2206.02512/) (2022)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations](https://mlsanity.com/paper/2308.13007/) (2023)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Referee: Towards reference-free cross-speaker style transfer with low-quality data for expressive speech synthesis](https://mlsanity.com/paper/2109.03439/) (2021)
- [Automatic Tuning of Loss Trade-offs without Hyper-parameter Search in End-to-End Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2305.16699/) (2023)
- [HIDFlowNet: A Flow-Based Deep Network for Hyperspectral Image Denoising](https://mlsanity.com/paper/2306.17797/) (2023)

## Abstract (arXiv)

Current two-stage TTS framework typically integrates an acoustic model with a vocoder -- the acoustic model predicts a low resolution intermediate representation such as Mel-spectrum while the vocoder generates waveform from the intermediate representation. Although the intermediate representation is served as a bridge, there still exists critical mismatch between the acoustic model and the vocoder as they are commonly separately learned and work on different distributions of representation, leading to inevitable artifacts in the synthesized speech. In this work, different from using pre-designed intermediate representation in most previous studies, we propose to use VAE combining with GAN to learn a latent representation directly from speech and then utilize a flow-based acoustic model to model the distribution of the latent representation from text. In this way, the mismatch problem is migrated as the two stages work on the same distribution. Results demonstrate that the flow-based acoustic model can exactly model the distribution of our learned speech representation and the proposed TTS framework, namely Glow-WaveGAN, can produce high fidelity speech outperforming the state-of-the-art GAN-based model.