mlsanityData to 7 Oct 2026

emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation

arXiv 2312.15185 ↗First posted 23 Dec 2023cs.CL cs.HC cs.MMcited by 196

Not on the map. Why?

Abstract (arXiv)

We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.

Key facts. Cited by 196 papers in the map (data through 7 Oct 2026). Builds on 29 of its 60 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

29 of 60 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
17 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  2. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  5. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  6. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
  7. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  8. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  9. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  10. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  11. 2018Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languagescited by 25
  12. 2019ShEMO -- A Large-Scale Validated Database for Persian Speech Emotion Detectioncited by 21
  13. 2021Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognitioncited by 14
  14. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  15. 2023DST: Deformable Speech Transformer for Emotion Recognitioncited by 11
  16. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  17. 2023Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 82; 2026: 85.

196 papers.

Sort:

By area:

  1. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  3. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  4. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  5. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  6. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  7. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  8. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  9. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
  10. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  11. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  12. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
38 more
  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  3. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  4. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  5. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  6. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  7. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  8. 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
  9. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  10. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  11. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 19
  12. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  13. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 17
  14. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  15. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  16. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  17. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  18. 2025OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academiacited by 13
  19. 2024DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animationcited by 12
  20. 2025MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animationcited by 12
  21. 2024Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selectioncited by 10
  22. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  23. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  24. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  25. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  26. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  27. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  28. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Datacited by 7
  29. 2025WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processingcited by 7
  30. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  31. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 7
  32. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  33. 2026Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Modelscited by 6
  34. 2026AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoningcited by 6
  35. 2024Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insightscited by 5
  36. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  37. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  38. 2025ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonalitycited by 5

50 most cited of 196 All 196 (JSON)