mlsanityData to 7 Oct 2026

Emotion Intensity and its Control for Emotional Voice Conversion

arXiv 2201.03967 ↗First posted 10 Jan 2022cs.SD cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In EVC, emotions are usually treated as discrete categories overlooking the fact that speech also conveys emotions with various intensity levels that the listener can perceive. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding. We further learn the actual emotion encoder from an emotion-labelled database and study the use of relative attributes to represent fine-grained emotion intensity. To ensure emotional intelligibility, we incorporate emotion classification loss and emotion embedding similarity loss into the training of the EVC network. As desired, the proposed network controls the fine-grained emotion intensity in the output speech. Through both objective and subjective evaluations, we validate the effectiveness of the proposed network for emotional expressiveness and emotion intensity control.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 45 of its 148 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

45 of 148 references.

References · 45

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  3. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  4. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  5. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  6. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  7. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  8. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
  9. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  10. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 77
  11. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  12. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
33 more
  1. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  2. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  3. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  4. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  5. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  6. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  7. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  8. 2020Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Datacited by 26
  9. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  10. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  11. 2018AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanismscited by 22
  12. 2020Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversioncited by 22
  13. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  14. 2019Modeling emotion in complex stories: the Stanford Emotional Narratives Datasetcited by 20
  15. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  16. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  17. 2020VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speechcited by 18
  18. 2021Towards Multi-Scale Style Control for Expressive Speech Synthesiscited by 18
  19. 2018Nonparallel Emotional Speech Conversioncited by 17
  20. 2021Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Trainingcited by 17
  21. 2019Emotional Voice Conversion using Multitask Learning with Text-to-speechcited by 15
  22. 2020Pretraining Techniques for Sequence-to-Sequence Voice Conversioncited by 13
  23. 2020Non-parallel Emotion Conversion using a Deep-Generative Hybrid Network and an Adversarial Pair Discriminatorcited by 12
  24. 2018Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervisioncited by 10
  25. 2018ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversioncited by 7
  26. 2021Non-autoregressive sequence-to-sequence voice conversioncited by 7
  27. 2020Many-to-Many Voice Transformer Networkcited by 6
  28. 2020Multi-speaker Emotion Conversion via Latent Variable Regularization and a Chained Encoder-Decoder-Predictor Networkcited by 6
  29. 2021EmoCat: Language-agnostic Emotional Voice Conversioncited by 6
  30. 2020Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglementcited by 5
  31. 2021FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversioncited by 5
  32. 2020Interactive Text-to-Speech System via Joint Style Analysiscited by 3
  33. 2020Recognition-Synthesis Based Non-Parallel Voice Conversion with Adversarial Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 10; 2025: 7; 2026: 3.

35 papers.

Sort:

By area:

  1. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  2. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  3. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  4. 2022Speech Synthesis with Mixed Emotionscited by 28
  5. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  6. 2023EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesiscited by 19
  7. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  8. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  9. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  10. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  11. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  12. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
23 more
  1. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  2. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
  3. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  4. 2022An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversioncited by 6
  5. 2022Emotion Selectable End-to-End Text-based Speech Editingcited by 3
  6. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  7. 2023Towards an Interpretable Representation of Speaker Identity via Perceptual Voice Qualitiescited by 3
  8. 2023Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversioncited by 3
  9. 2024Expressivity and Speech Synthesiscited by 3
  10. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3
  11. 2025ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speechcited by 3
  12. 2024Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglementcited by 2
  13. 2024Phonetic Enhanced Language Modeling for Text-to-Speech Synthesiscited by 2
  14. 2025Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversioncited by 2
  15. 2023PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Modelscited by 1
  16. 2024Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Modelcited by 1
  17. 2025Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosodycited by 1
  18. 2025Provable Speech Attributes Conversion via Latent Independencecited by 1
  19. 2025HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speechcited by 0
  20. 2025Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulationcited by 0
  21. 2026Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformerscited by 0
  22. 2026StyleStream: Real-Time Zero-Shot Voice Style Conversioncited by 0
  23. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0