# nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech

arXiv: https://arxiv.org/abs/2202.10712 · HTML: https://mlsanity.com/paper/2202.10712/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2022.

## Built on by (most cited first, 10 of 10)

- [Pre-Avatar: An Automatic Presentation Generation Framework Leveraging Talking Avatar](https://mlsanity.com/paper/2210.06877/) (2022)
- [USAT: A Universal Speaker-Adaptive Text-to-Speech Approach](https://mlsanity.com/paper/2404.18094/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Pose Guided Human Image Synthesis with Partially Decoupled GAN](https://mlsanity.com/paper/2210.03627/) (2022)
- [Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech](https://mlsanity.com/paper/2308.14909/) (2023)
- [Semi-Supervised Learning Based on Reference Model for Low-resource TTS](https://mlsanity.com/paper/2210.14723/) (2022)
- [Blur the Linguistic Boundary: Interpreting Chinese Buddhist Sutra in English via Neural Machine Translation](https://mlsanity.com/paper/2209.15164/) (2022)
- [Adapitch: Adaption Multi-Speaker Text-to-Speech Conditioned on Pitch Disentangling with Untranscribed Data](https://mlsanity.com/paper/2210.13803/) (2022)
- [MetaSpeech: Speech Effects Switch Along with Environment for Metaverse](https://mlsanity.com/paper/2210.13811/) (2022)
- [Neural Multi-Speaker Voice Cloning for Nepali in Low-Resource Settings](https://mlsanity.com/paper/2601.18694/) (2026)

## Abstract (arXiv)

Multi-speaker text-to-speech (TTS) using a few adaption data is a challenge in practical applications. To address that, we propose a zero-shot multi-speaker TTS, named nnSpeech, that could synthesis a new speaker voice without fine-tuning and using only one adaption utterance. Compared with using a speaker representation module to extract the characteristics of new speakers, our method bases on a speaker-guided conditional variational autoencoder and can generate a variable Z, which contains both speaker characteristics and content information. The latent variable Z distribution is approximated by another variable conditioned on reference mel-spectrogram and phoneme. Experiments on the English corpus, Mandarin corpus, and cross-dataset proves that our model could generate natural and similar speech with only one adaption speech.