mlsanityData to 7 Oct 2026

Emotional Voice Conversion: Theory, Databases and ESD

arXiv 2105.14762 ↗First posted 31 May 2021cs.CLcited by 113

Not on the map. Why?

Abstract (arXiv)

In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 hours of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 50 of its 201 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

50 of 201 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  7. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  8. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  9. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  10. 2018Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Datacited by 76
  11. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  12. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
38 more
  1. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
  2. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  3. 2017Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networkscited by 38
  4. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  5. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  6. 2017Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networkscited by 30
  7. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  8. 2019Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretrainingcited by 29
  9. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  10. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  11. 2020Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trendscited by 28
  12. 2020One Model, Many Languages: Meta-learning for Multilingual Text-to-Speechcited by 28
  13. 2020Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Datacited by 26
  14. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  15. 2018AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanismscited by 22
  16. 2020Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversioncited by 22
  17. 2019Exploring Transfer Learning for Low Resource Emotional TTScited by 21
  18. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  19. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  20. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  21. 2020VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speechcited by 18
  22. 2018Nonparallel Emotional Speech Conversioncited by 17
  23. 2020Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Datacited by 17
  24. 2021Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Trainingcited by 17
  25. 2019Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNetcited by 15
  26. 2019Emotional Voice Conversion using Multitask Learning with Text-to-speechcited by 15
  27. 2018High-quality nonparallel voice conversion based on cycle-consistent adversarial networkcited by 13
  28. 2020Non-parallel Emotion Conversion using a Deep-Generative Hybrid Network and an Adversarial Pair Discriminatorcited by 12
  29. 2019Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis through Audio Analysiscited by 8
  30. 2020Transferring Source Style in Non-Parallel Voice Conversioncited by 7
  31. 2019Reinforcement Learning Based Emotional Editing Constraint Conversation Generationcited by 6
  32. 2020Multi-speaker Emotion Conversion via Latent Variable Regularization and a Chained Encoder-Decoder-Predictor Networkcited by 6
  33. 2020VAW-GAN for Singing Voice Conversion with Non-parallel Training Datacited by 6
  34. 2021EmoCat: Language-agnostic Emotional Voice Conversioncited by 6
  35. 2020Spectrum and Prosody Conversion for Cross-lingual Voice Conversion with CycleGANcited by 5
  36. 2021Towards end-to-end F0 voice conversion based on Dual-GAN with convolutional wavelet kernelscited by 3
  37. 2020Multi-Target Emotional Voice Conversion With Neural Vocoderscited by 2
  38. 2020ICE-Talk: an Interface for a Controllable Expressive Talking Machinecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 20; 2024: 25; 2025: 31; 2026: 22.

113 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 210
  2. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
  3. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  4. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  5. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  6. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  7. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  8. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  9. 2021Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videoscited by 28
  10. 2022Speech Synthesis with Mixed Emotionscited by 28
  11. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  12. 2025Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliancecited by 23
38 more
  1. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  2. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  3. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  4. 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
  5. 2024Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generatorscited by 18
  6. 2022Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Unitscited by 15
  7. 2023ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Modelscited by 14
  8. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 14
  9. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  10. 2023Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs Using Dual Domain Adversarial Network & Virtual Domain Pairingcited by 9
  11. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  12. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  13. 2024Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognitioncited by 8
  14. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
  15. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  16. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8
  17. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  18. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  19. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generationcited by 7
  20. 2022An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversioncited by 6
  21. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversioncited by 6
  22. 2024Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognitioncited by 5
  23. 2024Anonymising Elderly and Pathological Speech: Voice Conversion Using DDSP and Query-by-Examplecited by 5
  24. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  25. 2025DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speechcited by 5
  26. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  27. 2026EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesiscited by 5
  28. 2021Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networkscited by 4
  29. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
  30. 2025Voice Cloning: Comprehensive Surveycited by 4
  31. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  32. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  33. 2025RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTScited by 4
  34. 2022SpeechEQ: Speech Emotion Recognition based on Multi-scale Unified Datasets and Multitask Learningcited by 3
  35. 2022Emotion Selectable End-to-End Text-based Speech Editingcited by 3
  36. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  37. 2023Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Datacited by 3
  38. 2023Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversioncited by 3

50 most cited of 113 All 113 (JSON)