mlsanityData to 7 Oct 2026

Exploring speech style spaces with language models: Emotional TTS without emotion labels

arXiv 2405.11413 ↗First posted 18 May 2024eess.AS cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Many frameworks for emotional text-to-speech (E-TTS) rely on human-annotated emotion labels that are often inaccurate and difficult to obtain. Learning emotional prosody implicitly presents a tough challenge due to the subjective nature of emotions. In this study, we propose a novel approach that leverages text awareness to acquire emotional styles without the need for explicit emotion labels or text prompts. We present TEMOTTS, a two-stage framework for E-TTS that is trained without emotion labels and is capable of inference without auxiliary inputs. Our proposed method performs knowledge transfer between the linguistic space learned by BERT and the emotional style space constructed by global style tokens. Our experimental results demonstrate the effectiveness of our proposed framework, showcasing improvements in emotional accuracy and naturalness. This is one of the first studies to leverage the emotional correlation between spoken content and expressive delivery for emotional TTS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

26 of 38 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  9. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  10. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  11. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  12. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
14 more
  1. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  2. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  3. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  4. 2021Expressive Text-to-Speech using Style Tagcited by 20
  5. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19
  6. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  7. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  8. 2021EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Modelcited by 18
  9. 2021Emotional Prosody Control for Speech Generationcited by 12
  10. 2021Speech BERT Embedding For Improving Prosody in Neural TTScited by 11
  11. 2022Self-supervised Context-aware Style Representation for Expressive Speech Synthesiscited by 10
  12. 2022Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systemscited by 9
  13. 2022Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTScited by 6
  14. 2023MSStyleTTS: Multi-Scale Style Modeling with Hierarchical Context Information for Expressive Speech Synthesiscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 7
  2. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3
  3. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  4. 2024Making Social Platforms Accessible: Emotion-Aware Speech Generation with Integrated Text Analysiscited by 0