Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Audio token modeling has become a powerful framework for speech synthesis, with two-stage approaches employing semantic tokens remaining prevalent. In this paper, we aim to simplify this process by introducing a semantic knowledge distillation method that enables high-quality speech generation in a single stage. Our proposed model improves speech quality, intelligibility, and speaker similarity compared to a single-stage baseline. Although two-stage systems still lead in intelligibility, our model significantly narrows the gap while delivering comparable speech quality. These findings showcase the potential of single-stage models to achieve efficient, high-quality TTS with a more compact and streamlined architecture.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 17 Sep 2024.
Builds on
33 of 42 references.
References · 33
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
21 more
- 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
- 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
- 2019Understanding and Improving Layer Normalizationcited by 213
- 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
- 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
- 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
- 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
- 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
- 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
- 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
- 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
- 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
- 2022SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguationcited by 14
- 2023Low-Resource Self-Supervised Learning with SSL-Enhanced TTScited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.