# Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning

arXiv: https://arxiv.org/abs/2309.04420 · HTML: https://mlsanity.com/paper/2309.04420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining](https://mlsanity.com/paper/1912.06813/) (2019)
- [CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion](https://mlsanity.com/paper/2010.11672/) (2020)
- [AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms](https://mlsanity.com/paper/1811.04076/) (2018)
- [Sequence-to-Sequence Acoustic Modeling for Voice Conversion](https://mlsanity.com/paper/1810.06865/) (2018)
- [Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data](https://mlsanity.com/paper/2009.14399/) (2020)
- [Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet](https://mlsanity.com/paper/1903.12389/) (2019)
- [Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Data](https://mlsanity.com/paper/2005.03295/) (2020)
- [Pretraining Techniques for Sequence-to-Sequence Voice Conversion](https://mlsanity.com/paper/2008.03088/) (2020)
- [RTFN: A Robust Temporal Feature Network for Time Series Classification](https://mlsanity.com/paper/2011.11829/) (2020)

## Abstract (arXiv)

Typically, voice conversion is regarded as an engineering problem with limited training data. The reliance on massive amounts of data hinders the practical applicability of deep learning approaches, which have been extensively researched in recent years. On the other hand, statistical methods are effective with limited data but have difficulties in modelling complex mapping functions. This paper proposes a voice conversion method that works with limited data and is based on stochastic variational deep kernel learning (SVDKL). At the same time, SVDKL enables the use of deep neural networks' expressive capability as well as the high flexibility of the Gaussian process as a Bayesian and non-parametric method. When the conventional kernel is combined with the deep neural network, it is possible to estimate non-smooth and more complex functions. Furthermore, the model's sparse variational Gaussian process solves the scalability problem and, unlike the exact Gaussian process, allows for the learning of a global mapping function for the entire acoustic space. One of the most important aspects of the proposed scheme is that the model parameters are trained using marginal likelihood optimization, which considers both data fitting and model complexity. Considering the complexity of the model reduces the amount of training data by increasing the resistance to overfitting. To evaluate the proposed scheme, we examined the model's performance with approximately 80 seconds of training data. The results indicated that our method obtained a higher mean opinion score, smaller spectral distortion, and better preference tests than the compared methods.