# Neural Multi-Speaker Voice Cloning for Nepali in Low-Resource Settings

arXiv: https://arxiv.org/abs/2601.18694 · HTML: https://mlsanity.com/paper/2601.18694/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 8 of 8)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech](https://mlsanity.com/paper/2202.10712/) (2022)
- [Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning](https://mlsanity.com/paper/2111.07218/) (2021)
- [Improve few-shot voice cloning using multi-modal learning](https://mlsanity.com/paper/2203.09708/) (2022)
- [Advancing Voice Cloning for Nepali: Leveraging Transfer Learning in a Low-Resource Language](https://mlsanity.com/paper/2408.10128/) (2024)

## Abstract (arXiv)

This research presents a few-shot voice cloning system for Nepali speakers, designed to synthesize speech in a specific speaker's voice from Devanagari text using minimal data. Voice cloning in Nepali remains largely unexplored due to its low-resource nature. To address this, we constructed separate datasets: untranscribed audio for training a speaker encoder and paired text-audio data for training a Tacotron2-based synthesizer. The speaker encoder, optimized with Generative End2End loss, generates embeddings that capture the speaker's vocal identity, validated through Uniform Manifold Approximation and Projection (UMAP) for dimension reduction visualizations. These embeddings are fused with Tacotron2's text embeddings to produce mel-spectrograms, which are then converted into audio using a WaveRNN vocoder. Audio data were collected from various sources, including self-recordings, and underwent thorough preprocessing for quality and alignment. Training was performed using mel and gate loss functions under multiple hyperparameter settings. The system effectively clones speaker characteristics even for unseen voices, demonstrating the feasibility of few-shot voice cloning for the Nepali language and establishing a foundation for personalized speech synthesis in low-resource scenarios.