mlsanityData to 7 Oct 2026

Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions

arXiv 2409.16681 ↗First posted 25 Sep 2024eess.AS cs.CL cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

Emotional text-to-speech (TTS) systems sturggle to capture the full spectrum of human emotions due to the inherent complexity of emotional expressions and the limited coverage of existing emotion labels. To address this, we propose a language model-based TTS framework that synthesizes speech across a broad range of emotional styles. Our approach enables flexible user control along three continuous dimensions - pleasure, arousal, and dominance (PAD). To enable this, we train an emotional dimension predictor that maps categorical emotion labels in speech datasets into the PAD space, grounded in established psychological research. Importantly, while the emotional dimension predictor leverages categorical labels, the TTS framework itself does not require explict emotion labels during training. Objective and subjective evaluations demonstrate that our framework effectively generates more expressive emotional styles and enhances both naturalness and diversity compared to baselines.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

32 of 47 references.

References · 32

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  6. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  7. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  8. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  9. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  10. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  11. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  12. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
20 more
  1. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  2. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  3. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  4. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  5. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
  6. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  7. 2022Speech Synthesis with Mixed Emotionscited by 28
  8. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  9. 2020Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversioncited by 22
  10. 2019Exploring Transfer Learning for Low Resource Emotional TTScited by 21
  11. 2023EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesiscited by 19
  12. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  13. 2021Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Trainingcited by 17
  14. 2022Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systemscited by 9
  15. 2024Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Modelcited by 8
  16. 2024Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesiscited by 6
  17. 2024Expressivity and Speech Synthesiscited by 3
  18. 2023Learning Arousal-Valence Representation from Categorical Emotion Labels of Speechcited by 2
  19. 2024Fine-Grained Quantitative Emotion Editing for Speech Generationcited by 2
  20. 2024Phonetic Enhanced Language Modeling for Text-to-Speech Synthesiscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 4.

8 papers.

  1. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  2. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  3. 2025DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speechcited by 5
  4. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  5. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 2
  6. 2026EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesiscited by 0
  7. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  8. 2026Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloningcited by 0