mlsanityData to 7 Oct 2026

Controlling Emotion in Text-to-Speech with Natural Language Prompts

arXiv 2406.06406 ↗First posted 10 Jun 2024cs.CL cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

37 of 48 references.

References · 37

  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  10. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  11. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  12. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
25 more
  1. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  2. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  3. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  4. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  5. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  6. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  7. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  8. 2016Yelp Dataset Challenge: Review Rating Predictioncited by 84
  9. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  10. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  11. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  12. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 33
  13. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
  14. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  15. 2021Expressive Text-to-Speech using Style Tagcited by 20
  16. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  17. 2022Avocodo: Generative Adversarial Network for Artifact-free Vocodercited by 18
  18. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17
  19. 2022Low-Resource Multilingual and Zero-Shot Multispeaker TTScited by 15
  20. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 14
  21. 2022Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speechcited by 13
  22. 2022Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTScited by 6
  23. 2023The IMS Toucan System for the Blizzard Challenge 2023cited by 6
  24. 2021Enhancing Word-Level Semantic Representation via Dependency Structure for Expressive Text-to-Speech Synthesiscited by 3
  25. 2022Contextual Expressive Text-to-Speechcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
  2. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 7
  3. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3
  4. 2025Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulationcited by 2
  5. 2024Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTScited by 1
  6. 2025AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesiscited by 1
  7. 2025EmoNews: A Spoken Dialogue System for Expressive News Conversationscited by 1
  8. 2026From Instruction to Output: The Role of Prompting in Modern NLGcited by 0