mlsanityData to 14 Oct 2026

An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era

arXiv 2210.03538 ↗First posted 6 Oct 2022cs.SD cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Speech is the fundamental mode of human communication, and its synthesis has long been a core priority in human-computer interaction research. In recent years, machines have managed to master the art of generating speech that is understandable by humans. But the linguistic content of an utterance encompasses only a part of its meaning. Affect, or expressivity, has the capacity to turn speech into a medium capable of conveying intimate thoughts, feelings, and emotions -- aspects that are essential for engaging and naturalistic interpersonal communication. While the goal of imparting expressivity to synthesised utterances has so far remained elusive, following recent advances in text-to-speech synthesis, a paradigm shift is well under way in the fields of affective speech synthesis and conversion as well. Deep learning, as the technology which underlies most of the recent advances in artificial intelligence, is spearheading these efforts. In the present overview, we outline ongoing trends and summarise state-of-the-art approaches in an attempt to provide a comprehensive overview of this exciting field.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 65 of its 207 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

65 of 207 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  8. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  12. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
38 more
  1. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  2. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  6. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  7. 2018Adversarial Audio Synthesiscited by 225
  8. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  9. 2021A Survey on Neural Speech Synthesiscited by 184
  10. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  11. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  12. 2016Unsupervised Cross-Domain Image Generationcited by 125
  13. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
  14. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  15. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  16. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 112
  17. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  18. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  19. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  20. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  21. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  22. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  23. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 32
  24. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  25. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  26. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  27. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  28. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  29. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  30. 2020Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Datacited by 26
  31. 2020Low-resource expressive text-to-speech using data augmentationcited by 26
  32. 2018AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanismscited by 22
  33. 2021Cross-speaker emotion disentangling and transfer for end-to-end speech synthesiscited by 21
  34. 2019Myers-Briggs Personality Classification and Personality-Specific Language Generation Using Pre-trained Language Modelscited by 20
  35. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19
  36. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  37. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  38. 2018Nonparallel Emotional Speech Conversioncited by 17

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 15; 2025: 7; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  2. 2025Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learningcited by 46
  3. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  4. 2024Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generatorscited by 18
  5. 2023U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speechcited by 9
  6. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  7. 2024SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detectioncited by 8
  8. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  9. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  10. 2024Affective Computing Has Changed: The Foundation Model Disruptioncited by 7
  11. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  12. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesiscited by 6
21 more
  1. 2025Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creationcited by 6
  2. 2023Learning Emotional Representations from Imbalanced Speech Data for Speech Emotion Recognition and Emotional Text-to-Speechcited by 4
  3. 2023Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Taskscited by 4
  4. 2024Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learningcited by 4
  5. 2024StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speechcited by 4
  6. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  7. 2024Expressivity and Speech Synthesiscited by 3
  8. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3
  9. 2024Audio-based Kinship Verification Using Age Domain Conversioncited by 3
  10. 2025BitTTS: Highly Compact Text-to-Speech Using 1.58-bit Quantization and Weight Indexingcited by 2
  11. 2025Bridging Minds and Machines: Toward an Integration of AI and Cognitive Sciencecited by 2
  12. 2026Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Modelscited by 2
  13. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 2
  14. 2022Automatic Emotion Modelling in Written Storiescited by 1
  15. 2024Non-verbal information in spontaneous speech -- towards a new framework of analysiscited by 1
  16. 2025Your voice is your voice: Supporting Self-expression through Speech Generation and LLMs in Augmented and Alternative Communicationcited by 1
  17. 2025Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphscited by 1
  18. 2025StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correctioncited by 1
  19. 2026Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluationcited by 0
  20. 2026EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generationcited by 0
  21. 2026Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detectioncited by 0