mlsanityData to 7 Oct 2026

Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability

arXiv 2104.01408 ↗First posted 3 Apr 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Emotional text-to-speech synthesis (ETTS) has seen much progress in recent years. However, the generated voice is often not perceptually identifiable by its intended emotion category. To address this problem, we propose a new interactive training paradigm for ETTS, denoted as i-ETTS, which seeks to directly improve the emotion discriminability by interacting with a speech emotion recognition (SER) model. Moreover, we formulate an iterative training strategy with reinforcement learning to ensure the quality of i-ETTS optimization. Experimental results demonstrate that the proposed i-ETTS outperforms the state-of-the-art baselines by rendering speech with more accurate emotion style. To our best knowledge, this is the first study of reinforcement learning in emotional text-to-speech synthesis.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 3 Apr 2021.

Builds on

21 of 33 references.

References · 21

  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  2. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  4. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  5. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  6. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  7. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  8. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  9. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  10. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  11. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  12. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
9 more
  1. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  2. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19
  3. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  4. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  5. 2019Teacher-Student Training for Robust Tacotron-based TTScited by 11
  6. 2018Reinforcement Learning Based Speech Enhancement for Robust Speech Recognitioncited by 10
  7. 2017Reinforcement Learning of Speech Recognition System Based on Policy Gradient and Hypothesis Selectioncited by 9
  8. 2018DNN-based Source Enhancement to Increase Objective Sound Quality Assessment Scorecited by 8
  9. 2020A Machine of Few Words -- Interactive Speaker Recognition with Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

19 papers.

Sort:

By area:

  1. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
  3. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  4. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  5. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  6. 2022Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiescited by 16
  7. 2021Fine-grained style control in Transformer-based Text-to-speech Synthesiscited by 15
  8. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  9. 2022Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlookcited by 10
  10. 2023Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrationscited by 8
  11. 2025YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidancecited by 5
  12. 2024Exploring speech style spaces with language models: Emotional TTS without emotion labelscited by 4
7 more
  1. 2021StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesiscited by 1
  2. 2026HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speechcited by 1
  3. 2022Computational Charisma -- A Brick by Brick Blueprint for Building Charismatic Artificial Intelligencecited by 0
  4. 2022MnTTS2: An Open-Source Multi-Speaker Mongolian Text-to-Speech Synthesis Datasetcited by 0
  5. 2023Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesiscited by 0
  6. 2025HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speechcited by 0
  7. 2026TokAN: Accent Normalization Using Self-Supervised Speech Tokenscited by 0