# Voice Cloning: a Multi-Speaker Text-to-Speech Synthesis Approach based on Transfer Learning

arXiv: https://arxiv.org/abs/2102.05630 · HTML: https://mlsanity.com/paper/2102.05630/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [1D Convolutional Neural Networks and Applications: A Survey](https://mlsanity.com/paper/1905.03554/) (2019)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [On Loss Functions for Deep Neural Networks in Classification](https://mlsanity.com/paper/1702.05659/) (2017)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [Fitting New Speakers Based on a Short Untranscribed Sample](https://mlsanity.com/paper/1802.06984/) (2018)

## Built on by (most cited first, 2 of 2)

- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equation](https://mlsanity.com/paper/2505.19273/) (2025)

## Abstract (arXiv)

Deep learning models are becoming predominant in many fields of machine learning. Text-to-Speech (TTS), the process of synthesizing artificial speech from text, is no exception. To this end, a deep neural network is usually trained using a corpus of several hours of recorded speech from a single speaker. Trying to produce the voice of a speaker other than the one learned is expensive and requires large effort since it is necessary to record a new dataset and retrain the model. This is the main reason why the TTS models are usually single speaker. The proposed approach has the goal to overcome these limitations trying to obtain a system which is able to model a multi-speaker acoustic space. This allows the generation of speech audio similar to the voice of different target speakers, even if they were not observed during the training phase.