# A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion

arXiv: https://arxiv.org/abs/2302.13678 · HTML: https://mlsanity.com/paper/2302.13678/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 33)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing](https://mlsanity.com/paper/2203.09690/) (2022)
- [Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks](https://mlsanity.com/paper/1704.00849/) (2017)
- [TraVeLGAN: Image-to-image Translation by Transformation Vector Learning](https://mlsanity.com/paper/1902.09631/) (2019)
- [F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder](https://mlsanity.com/paper/2004.07370/) (2020)
- [Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1711.11293/) (2017)
- [Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations](https://mlsanity.com/paper/1804.02812/) (2018)
- [Unsupervised Cross-Domain Singing Voice Conversion](https://mlsanity.com/paper/2008.02830/) (2020)
- [McGan: Mean and Covariance Feature Matching GAN](https://mlsanity.com/paper/1702.08398/) (2017)
- [FastSVC: Fast Cross-Domain Singing Voice Conversion with Feature-wise Linear Modulation](https://mlsanity.com/paper/2011.05731/) (2020)
- [Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders](https://mlsanity.com/paper/1912.02613/) (2019)
- [Audio query-based music source separation](https://mlsanity.com/paper/1908.06593/) (2019)
- [High-quality nonparallel voice conversion based on cycle-consistent adversarial network](https://mlsanity.com/paper/1804.00425/) (2018)
- [WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN](https://mlsanity.com/paper/1903.10729/) (2019)
- [MelGAN-VC: Voice Conversion and Audio Style Transfer on arbitrarily long samples using Spectrograms](https://mlsanity.com/paper/1910.03713/) (2019)
- [VAW-GAN for Singing Voice Conversion with Non-parallel Training Data](https://mlsanity.com/paper/2008.03992/) (2020)
- [A Hierarchical Speaker Representation Framework for One-shot Singing Voice Conversion](https://mlsanity.com/paper/2206.13762/) (2022)
- [Content Based Singing Voice Extraction From a Musical Mixture](https://mlsanity.com/paper/2002.04933/) (2020)

## Built on by (most cited first, 1 of 1)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)

## Abstract (arXiv)

Previous research has shown that established techniques for spoken voice conversion (VC) do not perform as well when applied to singing voice conversion (SVC). We propose an alternative loss component in a loss function that is otherwise well-established among VC tasks, which has been shown to improve our model's SVC performance. We first trained a singer identity embedding (SIE) network on mel-spectrograms of singer recordings to produce singer-specific variance encodings using contrastive learning. We subsequently trained a well-known autoencoder framework (AutoVC) conditioned on these SIEs, and measured differences in SVC performance when using different latent regressor loss components. We found that using this loss w.r.t. SIEs leads to better performance than w.r.t. bottleneck embeddings, where converted audio is more natural and specific towards target singers. The inclusion of this loss component has the advantage of explicitly forcing the network to reconstruct with timbral similarity, and also negates the effect of poor disentanglement in AutoVC's bottleneck embeddings. We demonstrate peculiar diversity between computational and human evaluations on singer-converted audio clips, which highlights the necessity of both. We also propose a pitch-matching mechanism between source and target singers to ensure these evaluations are not influenced by differences in pitch register.