# Sequence-to-Sequence Acoustic Modeling for Voice Conversion

arXiv: https://arxiv.org/abs/1810.06865 · HTML: https://mlsanity.com/paper/1810.06865/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2018.

## Built on by (most cited first, 20 of 20)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion](https://mlsanity.com/paper/2203.16705/) (2022)
- [MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames](https://mlsanity.com/paper/2102.12841/) (2021)
- [Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training](https://mlsanity.com/paper/2103.16809/) (2021)
- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Disentangleing Content and Fine-grained Prosody Information via Hybrid ASR Bottleneck Features for Voice Conversion](https://mlsanity.com/paper/2203.12813/) (2022)
- [Towards Improved Zero-shot Voice Conversion with Conditional DSVAE](https://mlsanity.com/paper/2205.05227/) (2022)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [Rhythm Modeling for Voice Conversion](https://mlsanity.com/paper/2307.06040/) (2023)
- [PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling](https://mlsanity.com/paper/2306.07489/) (2023)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [Phase-Aware Spoof Speech Detection Based on Res2Net with Phase Network](https://mlsanity.com/paper/2203.10793/) (2022)
- [Beyond Categorical Label Representations for Image Classification](https://mlsanity.com/paper/2104.02226/) (2021)
- [Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features](https://mlsanity.com/paper/2212.05805/) (2022)
- [Parallel and Limited Data Voice Conversion Using Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/2309.04420/) (2023)

## Abstract (arXiv)

In this paper, a neural network named Sequence-to-sequence ConvErsion NeTwork (SCENT) is presented for acoustic modeling in voice conversion. At training stage, a SCENT model is estimated by aligning the feature sequences of source and target speakers implicitly using attention mechanism. At conversion stage, acoustic features and durations of source utterances are converted simultaneously using the unified acoustic model. Mel-scale spectrograms are adopted as acoustic features which contain both excitation and vocal tract descriptions of speech signals. The bottleneck features extracted from source speech using an automatic speech recognition (ASR) model are appended as auxiliary input. A WaveNet vocoder conditioned on Mel-spectrograms is built to reconstruct waveforms from the outputs of the SCENT model. It is worth noting that our proposed method can achieve appropriate duration conversion which is difficult in conventional methods. Experimental results show that our proposed method obtained better objective and subjective performance than the baseline methods using Gaussian mixture models (GMM) and deep neural networks (DNN) as acoustic models. This proposed method also outperformed our previous work which achieved the top rank in Voice Conversion Challenge 2018. Ablation tests further confirmed the effectiveness of several components in our proposed method.