# Emotional Voice Conversion using Multitask Learning with Text-to-speech

arXiv: https://arxiv.org/abs/1911.06149 · HTML: https://mlsanity.com/paper/1911.06149/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training](https://mlsanity.com/paper/2103.16809/) (2021)
- [DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment](https://mlsanity.com/paper/2401.08095/) (2024)
- [An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversion](https://mlsanity.com/paper/2203.15873/) (2022)
- [Improve few-shot voice cloning using multi-modal learning](https://mlsanity.com/paper/2203.09708/) (2022)
- [Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion](https://mlsanity.com/paper/2312.17508/) (2023)
- [ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech](https://mlsanity.com/paper/2505.13805/) (2025)
- [NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion](https://mlsanity.com/paper/2511.00256/) (2025)
- [Multi-speaker Emotional Text-to-speech Synthesizer](https://mlsanity.com/paper/2112.03557/) (2021)
- [Boosting Star-GANs for Voice Conversion with Contrastive Discriminator](https://mlsanity.com/paper/2209.10088/) (2022)
- [Voice Cloning for Dysarthric Speech Synthesis: Addressing Data Scarcity in Speech-Language Pathology](https://mlsanity.com/paper/2503.01266/) (2025)

## Abstract (arXiv)

Voice conversion (VC) is a task to transform a person's voice to different style while conserving linguistic contents. Previous state-of-the-art on VC is based on sequence-to-sequence (seq2seq) model, which could mislead linguistic information. There was an attempt to overcome it by using textual supervision, it requires explicit alignment which loses the benefit of using seq2seq model. In this paper, a voice converter using multitask learning with text-to-speech (TTS) is presented. The embedding space of seq2seq-based TTS has abundant information on the text. The role of the decoder of TTS is to convert embedding space to speech, which is same to VC. In the proposed model, the whole network is trained to minimize loss of VC and TTS. VC is expected to capture more linguistic information and to preserve training stability by multitask learning. Experiments of VC were performed on a male Korean emotional text-speech dataset, and it is shown that multitask learning is helpful to keep linguistic contents in VC.