mlsanityData to 7 Oct 2026

Multimodal Emotion Recognition with High-level Speech and Text Features

arXiv 2111.10202 ↗First posted 29 Sep 2021eess.AS cs.CL cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

Automatic emotion recognition is one of the central concerns of the Human-Computer Interaction field as it can bridge the gap between humans and machines. Current works train deep learning models on low-level data representations to solve the emotion recognition task. Since emotion datasets often have a limited amount of data, these approaches may suffer from overfitting, and they may learn based on superficial cues. To address these issues, we propose a novel cross-representation speech model, inspired by disentanglement representation learning, to perform emotion recognition on wav2vec 2.0 speech features. We also train a CNN-based model to recognize emotions from text features extracted with Transformer-based models. We further combine the speech-based and text-based results with a score fusion approach. Our method is evaluated on the IEMOCAP dataset in a 4-class classification problem, and it surpasses current works on speech-only, text-only, and multimodal emotion recognition.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

21 of 42 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  8. 2015Listen, Attend and Spellcited by 542
  9. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  10. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  11. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  12. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
9 more
  1. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  2. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  3. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  4. 2018Exploring Disentangled Feature Representation Beyond Face Identificationcited by 36
  5. 2020Emotion recognition by fusing time synchronous and time asynchronous representationscited by 26
  6. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  7. 2019Speaker-invariant Affective Representation Learning via Adversarial Trainingcited by 14
  8. 2020Transformer based unsupervised pre-training for acoustic representation learningcited by 9
  9. 2020Advancing Multiple Instance Learning with Attention Modeling for Categorical Speech Emotion Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

20 papers.

Sort:

  1. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  2. 2022Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognitioncited by 9
  3. 2022Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priorscited by 8
  4. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  5. 2022Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based modelscited by 7
  6. 2022MMER: Multimodal Multi-task Learning for Speech Emotion Recognitioncited by 7
  7. 2024MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attentioncited by 6
  8. 2023Leveraging Label Information for Multimodal Emotion Recognitioncited by 3
  9. 2022VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognitioncited by 2
  10. 2022Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Datacited by 2
  11. 2022Distribution-based Emotion Recognition in Conversationcited by 2
  12. 2022Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine Learningcited by 2
8 more
  1. 2023Image Captioners Sometimes Tell More Than Images They Seecited by 2
  2. 2024Turbo your multi-modal classification with contrastive learningcited by 2
  3. 2022Affective Feedback Synthesis Towards Multimodal Text and Image Datacited by 1
  4. 2023A Comparison of Time-based Models for Multimodal Emotion Recognitioncited by 0
  5. 2024Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Datacited by 0
  6. 2025"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?cited by 0
  7. 2025Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardwarecited by 0
  8. 2026Evaluating multimodal emotion recognition in proactive conversational agents: A user studycited by 0