# A Hierarchical Speaker Representation Framework for One-shot Singing Voice Conversion

arXiv: https://arxiv.org/abs/2206.13762 · HTML: https://mlsanity.com/paper/2206.13762/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2022.

## Built on by (most cited first, 6 of 6)

- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [SongBsAb: A Dual Prevention Approach against Singing Voice Conversion based Illegal Song Covers](https://mlsanity.com/paper/2401.17133/) (2024)
- [DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching](https://mlsanity.com/paper/2508.05978/) (2025)
- [A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion](https://mlsanity.com/paper/2302.13678/) (2023)
- [Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training](https://mlsanity.com/paper/2309.00284/) (2023)
- [V.O.I.C.E (Voice, Ownership, Identity, Control, Expression): Risk Taxonomy of Synthetic Voice Generation From Empirical Data](https://mlsanity.com/paper/2604.24794/) (2026)

## Abstract (arXiv)

Typically, singing voice conversion (SVC) depends on an embedding vector, extracted from either a speaker lookup table (LUT) or a speaker recognition network (SRN), to model speaker identity. However, singing contains more expressive speaker characteristics than conversational speech. It is suspected that a single embedding vector may only capture averaged and coarse-grained speaker characteristics, which is insufficient for the SVC task. To this end, this work proposes a novel hierarchical speaker representation framework for SVC, which can capture fine-grained speaker characteristics at different granularity. It consists of an up-sampling stream and three down-sampling streams. The up-sampling stream transforms the linguistic features into audio samples, while one down-sampling stream of the three operates in the reverse direction. It is expected that the temporal statistics of each down-sampling block can represent speaker characteristics at different granularity, which will be engaged in the up-sampling blocks to enhance the speaker modeling. Experiment results verify that the proposed method outperforms both the LUT and SRN based SVC systems. Moreover, the proposed system supports the one-shot SVC with only a few seconds of reference audio.