mlsanityData to 7 Oct 2026

Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models

arXiv 2510.13293 ↗First posted 15 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

While Text-to-Speech (TTS) systems enable emotional control via natural-language instructions, expressiveness, naturalness, and speech quality degrade when the target emotion conflicts with the textual semantics. We propose a Cross-modal Consistency Guided Classifier-Free Guidance (CCG-CFG) method with dynamic scales based on the degree of inconsistency between the text emotion and the explicit speech emotion, replacing the dropout condition with the text emotion. We also distill the CCG-CFG guidance signal using a hard-sample mining strategy, improving the TTS model's emotional alignment capability. Evaluations on five emotional corpora and two TTS benchmarks show that our approaches applied to CosyVoice2 achieve up to a 12% absolute improvement in emotion-recognition accuracy and a 10% relative improvement in subjective scores, outperforming baselines including HierSpeech++, Qwen3-TTS, and original CosyVoice2, while preserving intelligibility, naturalness, and high speech quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

25 of 36 references.

References · 25

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  6. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  8. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  9. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  10. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  11. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  12. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
13 more
  1. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  2. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  3. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  4. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  5. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
  6. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 26
  7. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  8. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  9. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  10. 2025Classifier-free Guidance with Adaptive Scalingcited by 19
  11. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 11
  12. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  13. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesiscited by 6

Built on by

Not cited yet.