# Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

arXiv: https://arxiv.org/abs/2407.12229 · HTML: https://mlsanity.com/paper/2407.12229/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [Controllable Emotion Transfer For End-to-End Speech Synthesis](https://mlsanity.com/paper/2011.08679/) (2020)
- [Emotional End-to-End Neural Speech Synthesizer](https://mlsanity.com/paper/1711.05447/) (2017)
- [MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis](https://mlsanity.com/paper/2201.06460/) (2022)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition](https://mlsanity.com/paper/2010.13350/) (2020)
- [EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis](https://mlsanity.com/paper/2306.00648/) (2023)
- [Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like](https://mlsanity.com/paper/2402.07383/) (2024)
- [JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions](https://mlsanity.com/paper/2310.06072/) (2023)
- [ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesis](https://mlsanity.com/paper/2401.08166/) (2024)
- [QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis](https://mlsanity.com/paper/2303.07682/) (2023)
- [Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS](https://mlsanity.com/paper/2207.06000/) (2022)
- [DiariST: Streaming Speech Translation with Speaker Diarization](https://mlsanity.com/paper/2309.08007/) (2023)

## Built on by (most cited first, 19 of 19)

- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations](https://mlsanity.com/paper/2508.04195/) (2025)
- [NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech](https://mlsanity.com/paper/2507.13155/) (2025)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions](https://mlsanity.com/paper/2507.10859/) (2025)
- [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025)
- [MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling](https://mlsanity.com/paper/2505.15772/) (2025)
- [EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis](https://mlsanity.com/paper/2608.23791/) (2026)
- [NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion](https://mlsanity.com/paper/2511.00256/) (2025)
- [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025)
- [The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation](https://mlsanity.com/paper/2604.26347/) (2026)
- [Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models](https://mlsanity.com/paper/2605.27376/) (2026)
- [VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation](https://mlsanity.com/paper/2608.13613/) (2026)
- [Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction](https://mlsanity.com/paper/2608.28970/) (2026)
- [HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech](https://mlsanity.com/paper/2509.21676/) (2025)
- [MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation](https://mlsanity.com/paper/2604.17435/) (2026)
- [Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS](https://mlsanity.com/paper/2606.09837/) (2026)

## Abstract (arXiv)

People change their tones of voice, often accompanied by nonverbal vocalizations (NVs) such as laughter and cries, to convey rich emotions. However, most text-to-speech (TTS) systems lack the capability to generate speech with rich emotions, including NVs. This paper introduces EmoCtrl-TTS, an emotion-controllable zero-shot TTS that can generate highly emotional speech with NVs for any speaker. EmoCtrl-TTS leverages arousal and valence values, as well as laughter embeddings, to condition the flow-matching-based zero-shot TTS. To achieve high-quality emotional speech generation, EmoCtrl-TTS is trained using more than 27,000 hours of expressive data curated based on pseudo-labeling. Comprehensive evaluations demonstrate that EmoCtrl-TTS excels in mimicking the emotions of audio prompts in speech-to-speech translation scenarios. We also show that EmoCtrl-TTS can capture emotion changes, express strong emotions, and generate various NVs in zero-shot TTS. See https://aka.ms/emoctrl-tts for demo samples.