# Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features

arXiv: https://arxiv.org/abs/2502.04722 · HTML: https://mlsanity.com/paper/2502.04722/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion](https://mlsanity.com/paper/2105.13871/) (2021)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [Boosting Self-Supervised Embeddings for Speech Enhancement](https://mlsanity.com/paper/2204.03339/) (2022)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders](https://mlsanity.com/paper/1912.02613/) (2019)
- [DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System](https://mlsanity.com/paper/2008.03009/) (2020)
- [PPG-based singing voice conversion with adversarial representation learning](https://mlsanity.com/paper/2010.14804/) (2020)
- [Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversion](https://mlsanity.com/paper/2203.12813/) (2022)
- [Hierarchical disentangled representation learning for singing voice conversion](https://mlsanity.com/paper/2101.06842/) (2021)
- [Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding](https://mlsanity.com/paper/2110.04754/) (2021)
- [Self-Supervised Representations for Singing Voice Conversion](https://mlsanity.com/paper/2303.12197/) (2023)
- [VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modeling](https://mlsanity.com/paper/2310.02802/) (2023)
- [RobustSVC: HuBERT-based Melody Extractor and Adversarial Learning for Robust Singing Voice Conversion](https://mlsanity.com/paper/2409.06237/) (2024)
- [Adversarial Speaker Disentanglement Using Unannotated External Data for Self-supervised Representation Based Voice Conversion](https://mlsanity.com/paper/2305.09167/) (2023)

## Built on by (most cited first, 1 of 1)

- [YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases](https://mlsanity.com/paper/2512.04793/) (2025)

## Abstract (arXiv)

Melody preservation is crucial in singing voice conversion (SVC). However, in many scenarios, audio is often accompanied with background music (BGM), which can cause audio distortion and interfere with the extraction of melody and other key features, significantly degrading SVC performance. Previous methods have attempted to address this by using more robust neural network-based melody extractors, but their performance drops sharply in the presence of complex accompaniment. Other approaches involve performing source separation before conversion, but this often introduces noticeable artifacts, leading to a significant drop in conversion quality and increasing the user's operational costs. To address these issues, we introduce a novel SVC method that uses self-supervised representation-based melody features to improve melody modeling accuracy in the presence of BGM. In our experiments, we compare the effectiveness of different self-supervised learning (SSL) models for melody extraction and explore for the first time how SSL benefits the task of melody extraction. The experimental results demonstrate that our proposed SVC model significantly outperforms existing baseline methods in terms of melody accuracy and shows higher similarity and naturalness in both subjective and objective evaluations across noisy and clean audio environments.