# Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer

arXiv: https://arxiv.org/abs/2208.12410 · HTML: https://mlsanity.com/paper/2208.12410/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [BEGAN: Boundary Equilibrium Generative Adversarial Networks](https://mlsanity.com/paper/1703.10717/) (2017)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [A Universal Music Translation Network](https://mlsanity.com/paper/1805.07848/) (2018)
- [NHSS: A Speech and Singing Parallel Database](https://mlsanity.com/paper/2012.00337/) (2020)
- [Conditional End-to-End Audio Transforms](https://mlsanity.com/paper/1804.00047/) (2018)
- [Vocal melody extraction using patch-based CNN](https://mlsanity.com/paper/1804.09202/) (2018)
- [End-to-end lyrics Recognition with Voice to Singing Style Transfer](https://mlsanity.com/paper/2102.08575/) (2021)
- [Speech-to-Singing Conversion in an Encoder-Decoder Framework](https://mlsanity.com/paper/2002.06595/) (2020)
- [Speech-to-Singing Conversion based on Boundary Equilibrium GAN](https://mlsanity.com/paper/2005.13835/) (2020)
- [Singing Style Transfer Using Cycle-Consistent Boundary Equilibrium Generative Adversarial Networks](https://mlsanity.com/paper/1807.02254/) (2018)

## Built on by (most cited first, 2 of 2)

- [Hierarchical Diffusion Models for Singing Voice Neural Vocoder](https://mlsanity.com/paper/2210.07508/) (2022)
- [Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement](https://mlsanity.com/paper/2401.04511/) (2024)

## Abstract (arXiv)

In this paper, we propose a model to perform style transfer of speech to singing voice. Contrary to the previous signal processing-based methods, which require high-quality singing templates or phoneme synchronization, we explore a data-driven approach for the problem of converting natural speech to singing voice. We develop a novel neural network architecture, called SymNet, which models the alignment of the input speech with the target melody while preserving the speaker identity and naturalness. The proposed SymNet model is comprised of symmetrical stack of three types of layers - convolutional, transformer, and self-attention layers. The paper also explores novel data augmentation and generative loss annealing methods to facilitate the model training. Experiments are performed on the NUS and NHSS datasets which consist of parallel data of speech and singing voice. In these experiments, we show that the proposed SymNet model improves the objective reconstruction quality significantly over the previously published methods and baseline architectures. Further, a subjective listening test confirms the improved quality of the audio obtained using the proposed approach (absolute improvement of 0.37 in mean opinion score measure over the baseline system).