# Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data

arXiv: https://arxiv.org/abs/2002.00198 · HTML: https://mlsanity.com/paper/2002.00198/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2020.

## Built on by (most cited first, 26 of 26)

- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training](https://mlsanity.com/paper/2103.16809/) (2021)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs Using Dual Domain Adversarial Network & Virtual Domain Pairing](https://mlsanity.com/paper/2302.10536/) (2023)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data](https://mlsanity.com/paper/2309.07828/) (2023)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversion](https://mlsanity.com/paper/2203.15873/) (2022)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networks](https://mlsanity.com/paper/2110.01164/) (2021)
- [CycleTransGAN-EVC: A CycleGAN-based Emotional Voice Conversion Model with Transformer](https://mlsanity.com/paper/2111.15159/) (2021)
- [Towards end-to-end F0 voice conversion based on Dual-GAN with convolutional wavelet kernels](https://mlsanity.com/paper/2104.07283/) (2021)
- [Improving Speech Emotion Recognition with Unsupervised Speaking Style Transfer](https://mlsanity.com/paper/2211.08843/) (2022)
- [Emotion Selectable End-to-End Text-based Speech Editing](https://mlsanity.com/paper/2212.10191/) (2022)
- [In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesis](https://mlsanity.com/paper/2306.01916/) (2023)
- [Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion](https://mlsanity.com/paper/2312.17508/) (2023)
- [ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech](https://mlsanity.com/paper/2505.13805/) (2025)
- [Provable Speech Attributes Conversion via Latent Independence](https://mlsanity.com/paper/2510.05191/) (2025)
- [Re-ENACT: Reinforcement Learning for Emotional Speech Generation using Actor-Critic Strategy](https://mlsanity.com/paper/2408.01892/) (2024)
- [Improving speaker verification robustness with synthetic emotional utterances](https://mlsanity.com/paper/2412.00319/) (2024)
- [Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset](https://mlsanity.com/paper/2505.20341/) (2025)

## Abstract (arXiv)

Emotional voice conversion aims to convert the spectrum and prosody to change the emotional patterns of speech, while preserving the speaker identity and linguistic content. Many studies require parallel speech data between different emotional patterns, which is not practical in real life. Moreover, they often model the conversion of fundamental frequency (F0) with a simple linear transform. As F0 is a key aspect of intonation that is hierarchical in nature, we believe that it is more adequate to model F0 in different temporal scales by using wavelet transform. We propose a CycleGAN network to find an optimal pseudo pair from non-parallel training data by learning forward and inverse mappings simultaneously using adversarial and cycle-consistency losses. We also study the use of continuous wavelet transform (CWT) to decompose F0 into ten temporal scales, that describes speech prosody at different time resolution, for effective F0 conversion. Experimental results show that our proposed framework outperforms the baselines both in objective and subjective evaluations.