# Speaker Generation

arXiv: https://arxiv.org/abs/2111.05095 · HTML: https://mlsanity.com/paper/2111.05095/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis](https://mlsanity.com/paper/1910.10288/) (2019)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis](https://mlsanity.com/paper/1808.01410/) (2018)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization](https://mlsanity.com/paper/2002.01953/) (2020)
- [Semi-Supervised Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1910.01709/) (2019)
- [Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis](https://mlsanity.com/paper/1906.03402/) (2019)
- [Improved generator objectives for GANs](https://mlsanity.com/paper/1612.02780/) (2016)
- [Properties of f-divergences and f-GAN training](https://mlsanity.com/paper/2009.00757/) (2020)

## Built on by (most cited first, 14 of 14)

- [PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions](https://mlsanity.com/paper/2309.08140/) (2023)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [Evaluating and reducing the distance between synthetic and real speech distributions](https://mlsanity.com/paper/2211.16049/) (2022)
- [VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesis](https://mlsanity.com/paper/2403.00529/) (2024)
- [Stutter-TTS: Controlled Synthesis and Improved Recognition of Stuttered Speech](https://mlsanity.com/paper/2211.09731/) (2022)
- [Self-supervised learning for robust voice cloning](https://mlsanity.com/paper/2204.03421/) (2022)
- [Residual Information in Deep Speaker Embedding Architectures](https://mlsanity.com/paper/2302.02742/) (2023)
- [Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/2209.12549/) (2022)
- [Creating New Voices using Normalizing Flows](https://mlsanity.com/paper/2312.14569/) (2023)
- [R-MelNet: Reduced Mel-Spectral Modeling for Neural TTS](https://mlsanity.com/paper/2206.15276/) (2022)
- [Generating Multilingual Gender-Ambiguous Text-to-Speech Voices](https://mlsanity.com/paper/2211.00375/) (2022)
- [SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy](https://mlsanity.com/paper/2507.23292/) (2025)
- [NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization](https://mlsanity.com/paper/2607.03985/) (2026)
- [VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS](https://mlsanity.com/paper/2610.14694/) (2026)

## Abstract (arXiv)

This work explores the task of synthesizing speech in nonexistent human-sounding voices. We call this task "speaker generation", and present TacoSpawn, a system that performs competitively at this task. TacoSpawn is a recurrent attention-based text-to-speech model that learns a distribution over a speaker embedding space, which enables sampling of novel and diverse speakers. Our method is easy to implement, and does not require transfer learning from speaker ID systems. We present objective and subjective metrics for evaluating performance on this task, and demonstrate that our proposed objective metrics correlate with human perception of speaker similarity. Audio samples are available on our demo page.