Multi-modal Adversarial Training for Zero-Shot Voice Cloning
Not on the map. Why?
Abstract (arXiv)
A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity. Audio examples from our system are available online.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 27 references that are in the map. First posted on arXiv 28 Aug 2024.
Builds on
25 of 27 references.
References · 25
- 2017Attention Is All You Needcited by 59,932
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
- 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
- 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
- 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
- 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
- 2021A Survey on Neural Speech Synthesiscited by 182
- 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
13 more
- 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
- 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
- 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
- 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
- 2022Revisiting Over-Smoothness in Text to Speechcited by 31
- 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
- 2020Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesiscited by 28
- 2023VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Designcited by 25
- 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
- 2021GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesiscited by 18
- 2020GAZEV: GAN-Based Zero-Shot Voice Conversion over Non-parallel Speech Corpuscited by 10
- 2018Reducing over-smoothness in speech synthesis using Generative Adversarial Networkscited by 4
- 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Modelcited by 3
Built on by
Citing papers per year: 2024: 2.
2 papers.