The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation
Not on the map. Why?
Abstract (arXiv)
Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring emotional prosody transfer. To quantify this, the field widely relies on emotion similarity between reference and generated samples. This approach computes cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations. We challenge this assumption through controlled adversarial tasks and human alignment tests. Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation. Representational limitations cause linguistic and speaker interference to overshadow emotional features, degrading discriminative ability. Consequently, the metric misaligns with human perception. This acoustic vulnerability reveals it rewards acoustic mimicry over genuine emotional synthesis.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
44 of 56 references.
References · 44
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
- 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
- 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
- 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
- 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
- 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
32 more
- 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
- 2026Qwen3-TTS Technical Reportcited by 106
- 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
- 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
- 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
- 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
- 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
- 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
- 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
- 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
- 2024Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognitioncited by 19
- 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
- 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
- 2024On the social bias of speech self-supervised modelscited by 16
- 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
- 2025The MSP-Podcast Corpuscited by 14
- 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
- 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
- 2025Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesiscited by 7
- 2022Large Raw Emotional Dataset with Aggregation Mechanismcited by 5
- 2025BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insightscited by 5
- 2025DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speechcited by 5
- 2025HD-PPT: Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTScited by 4
- 2025ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speechcited by 3
- 2022Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processingcited by 2
- 2025Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulationcited by 2
- 2025Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimizationcited by 2
- 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
- 2025Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spottingcited by 1
- 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNetcited by 1
- 2025Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosodycited by 1
- 2025Perturbation Self-Supervised Representations for Cross-Lingual Emotion TTS: Stage-Wise Modeling of Emotion and Speakercited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.