# Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning

arXiv: https://arxiv.org/abs/2109.11115 · HTML: https://mlsanity.com/paper/2109.11115/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2021.

## Built on by (most cited first, 5 of 5)

- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [ProSE: Diffusion Priors for Speech Enhancement](https://mlsanity.com/paper/2503.06375/) (2025)
- [The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge](https://mlsanity.com/paper/2410.23815/) (2024)
- [Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation](https://mlsanity.com/paper/2408.10557/) (2024)
- [Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencing](https://mlsanity.com/paper/2512.13904/) (2025)

## Abstract (arXiv)

One-shot voice cloning aims to transform speaker voice and speaking style in speech synthesized from a text-to-speech (TTS) system, where only a shot recording from the target reference speech can be used. Out-of-domain transfer is still a challenging task, and one important aspect that impacts the accuracy and similarity of synthetic speech is the conditional representations carrying speaker or style cues extracted from the limited references. In this paper, we present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.