mlsanityData to 7 Oct 2026

Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

arXiv 2407.12229 ↗First posted 17 Jul 2024eess.AS cs.AI eess.SPcited by 19

Not on the map. Why?

Abstract (arXiv)

People change their tones of voice, often accompanied by nonverbal vocalizations (NVs) such as laughter and cries, to convey rich emotions. However, most text-to-speech (TTS) systems lack the capability to generate speech with rich emotions, including NVs. This paper introduces EmoCtrl-TTS, an emotion-controllable zero-shot TTS that can generate highly emotional speech with NVs for any speaker. EmoCtrl-TTS leverages arousal and valence values, as well as laughter embeddings, to condition the flow-matching-based zero-shot TTS. To achieve high-quality emotional speech generation, EmoCtrl-TTS is trained using more than 27,000 hours of expressive data curated based on pseudo-labeling. Comprehensive evaluations demonstrate that EmoCtrl-TTS excels in mimicking the emotions of audio prompts in speech-to-speech translation scenarios. We also show that EmoCtrl-TTS can capture emotion changes, express strong emotions, and generate various NVs in zero-shot TTS. See https://aka.ms/emoctrl-tts for demo samples.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  12. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
20 more
  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  2. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  3. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 165
  4. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  5. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  6. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  7. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  8. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  9. 2022Speech Synthesis with Mixed Emotionscited by 28
  10. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  11. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  12. 2023EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesiscited by 19
  13. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  14. 2023JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressionscited by 10
  15. 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesiscited by 9
  16. 2023QI-TTS: Questioning Intonation Control for Emotional Speech Synthesiscited by 8
  17. 2022Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTScited by 6
  18. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  19. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  20. 2024Total-Duration-Aware Duration Modeling for Text-to-Speech Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
  2. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  3. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  4. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizationscited by 14
  5. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  6. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  7. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  8. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  9. 2025MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labelingcited by 3
  10. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  11. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 2
  12. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
7 more
  1. 2026The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluationcited by 1
  2. 2026Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Modelscited by 1
  3. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentationcited by 1
  4. 2025HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speechcited by 0
  5. 2026MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translationcited by 0
  6. 2026Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTScited by 0
  7. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion ↗
  2. 2024Hierarchical Control of Emotion Rendering in Speech Synthesis
  3. 2023ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models
  4. 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
  5. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling
  6. 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
  7. 2026DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
  8. 2025EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering