# SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement

arXiv: https://arxiv.org/abs/2407.07728 · HTML: https://mlsanity.com/paper/2407.07728/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis](https://mlsanity.com/paper/1711.00354/) (2017)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder](https://mlsanity.com/paper/2004.07370/) (2020)
- [RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music](https://mlsanity.com/paper/2306.15412/) (2023)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [Unsupervised Cross-Domain Singing Voice Conversion](https://mlsanity.com/paper/2008.02830/) (2020)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders](https://mlsanity.com/paper/1912.02613/) (2019)
- [PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network](https://mlsanity.com/paper/1912.01852/) (2019)
- [JVS-MuSiC: Japanese multispeaker singing-voice corpus](https://mlsanity.com/paper/2001.07044/) (2020)
- [PPG-based singing voice conversion with adversarial representation learning](https://mlsanity.com/paper/2010.14804/) (2020)
- [Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis](https://mlsanity.com/paper/2205.04029/) (2022)
- [PJS: phoneme-balanced Japanese singing voice corpus](https://mlsanity.com/paper/2006.02959/) (2020)
- [STFT spectral loss for training a neural speech waveform model](https://mlsanity.com/paper/1810.11945/) (2018)
- [Improving Adversarial Waveform Generation based Singing Voice Conversion with Harmonic Signals](https://mlsanity.com/paper/2201.10130/) (2022)

## Built on by (most cited first, 5 of 5)

- [Marco-Voice Technical Report](https://mlsanity.com/paper/2508.02038/) (2025)
- [HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios](https://mlsanity.com/paper/2511.08496/) (2025)
- [R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversion](https://mlsanity.com/paper/2510.20677/) (2025)
- [Multi-Accent Mandarin Dry-Vocal Singing Dataset: Benchmark for Singing Accent Recognition](https://mlsanity.com/paper/2512.07005/) (2025)
- [Singing Timbre Popularity Assessment Based on Multimodal Large Foundation Model](https://mlsanity.com/paper/2512.06999/) (2025)

## Abstract (arXiv)

Singing voice conversion (SVC) aims to convert a singer's voice to another singer's from a reference audio while keeping the original semantics. However, existing SVC methods can hardly perform zero-shot due to incomplete feature disentanglement or dependence on the speaker look-up table. We propose the first open-source high-quality zero-shot SVC model SaMoye that can convert singing to human and non-human timbre. SaMoye disentangles the singing voice's features into content, timbre, and pitch features, where we combine multiple ASR models and compress the content features to reduce timbre leaks. Besides, we enhance the timbre features by unfreezing the speaker encoder and mixing the speaker embedding with top-3 similar speakers. We also establish an unparalleled large-scale dataset to guarantee zero-shot performance, which comprises more than 1,815 hours of pure singing voice and 6,367 speakers. We conduct objective and subjective experiments to find that SaMoye outperforms other models in zero-shot SVC tasks even under extreme conditions like converting singing to animals' timbre. The code and weight of SaMoye are available on https://github.com/CarlWangChina/SaMoye-SVC. The weights, code, dataset, and documents of SaMoye are publicly available on urlhttps://github.com/CarlWangChina/SaMoye-SVC.