# DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders

arXiv: https://arxiv.org/abs/2207.04646 · HTML: https://mlsanity.com/paper/2207.04646/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2022.

## Built on by (most cited first, 15 of 15)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias](https://mlsanity.com/paper/2306.03509/) (2023)
- [Where are we in audio deepfake detection? A systematic analysis over generative and detection models](https://mlsanity.com/paper/2410.04324/) (2024)
- [HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer](https://mlsanity.com/paper/2307.16171/) (2023)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Zero-Shot Text-to-Speech for Vietnamese](https://mlsanity.com/paper/2506.01322/) (2025)
- [Next Tokens Denoising for Speech Synthesis](https://mlsanity.com/paper/2507.22746/) (2025)
- [LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space](https://mlsanity.com/paper/2603.29339/) (2026)
- [Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis](https://mlsanity.com/paper/2507.00227/) (2025)
- [DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer](https://mlsanity.com/paper/2305.19567/) (2023)
- [Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study](https://mlsanity.com/paper/2406.04633/) (2024)
- [Generative Adversarial Training for Text-to-Speech Synthesis Based on Raw Phonetic Input and Explicit Prosody Modelling](https://mlsanity.com/paper/2310.09636/) (2023)
- [Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation](https://mlsanity.com/paper/2603.01565/) (2026)

## Abstract (arXiv)

Current text to speech (TTS) systems usually leverage a cascaded acoustic model and vocoder pipeline with mel-spectrograms as the intermediate representations, which suffer from two limitations: 1) the acoustic model and vocoder are separately trained instead of jointly optimized, which incurs cascaded errors; 2) the intermediate speech representations (e.g., mel-spectrogram) are pre-designed and lose phase information, which are sub-optimal. To solve these problems, in this paper, we develop DelightfulTTS 2, a new end-to-end speech synthesis system with automatically learned speech representations and jointly optimized acoustic model and vocoder. Specifically, 1) we propose a new codec network based on vector-quantized auto-encoders with adversarial training (VQ-GAN) to extract intermediate frame-level speech representations (instead of traditional representations like mel-spectrograms) and reconstruct speech waveform; 2) we jointly optimize the acoustic model (based on DelightfulTTS) and the vocoder (the decoder of VQ-GAN), with an auxiliary loss on the acoustic model to predict intermediate speech representations. Experiments show that DelightfulTTS 2 achieves a CMOS gain +0.14 over DelightfulTTS, and more method analyses further verify the effectiveness of the developed system.