# Improve few-shot voice cloning using multi-modal learning

arXiv: https://arxiv.org/abs/2203.09708 · HTML: https://mlsanity.com/paper/2203.09708/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization](https://mlsanity.com/paper/2002.01953/) (2020)
- [Fitting New Speakers Based on a Short Untranscribed Sample](https://mlsanity.com/paper/1802.06984/) (2018)
- [Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning](https://mlsanity.com/paper/1910.12729/) (2019)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)
- [Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet](https://mlsanity.com/paper/1903.12389/) (2019)
- [Emotional Voice Conversion using Multitask Learning with Text-to-speech](https://mlsanity.com/paper/1911.06149/) (2019)
- [NAUTILUS: a Versatile Voice Cloning System](https://mlsanity.com/paper/2005.11004/) (2020)
- [The NeteaseGames System for Voice Conversion Challenge 2020 with Vector-quantization Variational Autoencoder and WaveNet](https://mlsanity.com/paper/2010.07630/) (2020)

## Built on by (most cited first, 3 of 3)

- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Advancing Voice Cloning for Nepali: Leveraging Transfer Learning in a Low-Resource Language](https://mlsanity.com/paper/2408.10128/) (2024)
- [Neural Multi-Speaker Voice Cloning for Nepali in Low-Resource Settings](https://mlsanity.com/paper/2601.18694/) (2026)

## Abstract (arXiv)

Recently, few-shot voice cloning has achieved a significant improvement. However, most models for few-shot voice cloning are single-modal, and multi-modal few-shot voice cloning has been understudied. In this paper, we propose to use multi-modal learning to improve the few-shot voice cloning performance. Inspired by the recent works on unsupervised speech representation, the proposed multi-modal system is built by extending Tacotron2 with an unsupervised speech representation module. We evaluate our proposed system in two few-shot voice cloning scenarios, namely few-shot text-to-speech(TTS) and voice conversion(VC). Experimental results demonstrate that the proposed multi-modal learning can significantly improve the few-shot voice cloning performance over their counterpart single-modal systems.