# Multimodal Emotion Recognition with High-level Speech and Text Features

arXiv: https://arxiv.org/abs/2111.10202 · HTML: https://mlsanity.com/paper/2111.10202/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2008.06682/) (2020)
- [Exploring Disentangled Feature Representation Beyond Face Identification](https://mlsanity.com/paper/1804.03487/) (2018)
- [Emotion recognition by fusing time synchronous and time asynchronous representations](https://mlsanity.com/paper/2010.14102/) (2020)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [Speaker-invariant Affective Representation Learning via Adversarial Training](https://mlsanity.com/paper/1911.01533/) (2019)
- [Transformer based unsupervised pre-training for acoustic representation learning](https://mlsanity.com/paper/2007.14602/) (2020)
- [Advancing Multiple Instance Learning with Attention Modeling for Categorical Speech Emotion Recognition](https://mlsanity.com/paper/2008.06667/) (2020)

## Built on by (most cited first, 20 of 20)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors](https://mlsanity.com/paper/2203.04443/) (2022)
- [Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.09856/) (2023)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [MFHCA: Enhancing Speech Emotion Recognition Via Multi-Spatial Fusion and Hierarchical Cooperative Attention](https://mlsanity.com/paper/2404.13509/) (2024)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition](https://mlsanity.com/paper/2208.11450/) (2022)
- [Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Data](https://mlsanity.com/paper/2208.11868/) (2022)
- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)
- [Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine Learning](https://mlsanity.com/paper/2211.09273/) (2022)
- [Image Captioners Sometimes Tell More Than Images They See](https://mlsanity.com/paper/2305.02932/) (2023)
- [Turbo your multi-modal classification with contrastive learning](https://mlsanity.com/paper/2409.09282/) (2024)
- [Affective Feedback Synthesis Towards Multimodal Text and Image Data](https://mlsanity.com/paper/2203.12692/) (2022)
- [A Comparison of Time-based Models for Multimodal Emotion Recognition](https://mlsanity.com/paper/2306.13076/) (2023)
- [Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Data](https://mlsanity.com/paper/2402.07640/) (2024)
- ["Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?](https://mlsanity.com/paper/2501.04138/) (2025)
- [Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardware](https://mlsanity.com/paper/2510.18036/) (2025)
- [Evaluating multimodal emotion recognition in proactive conversational agents: A user study](https://mlsanity.com/paper/2605.20200/) (2026)

## Abstract (arXiv)

Automatic emotion recognition is one of the central concerns of the Human-Computer Interaction field as it can bridge the gap between humans and machines. Current works train deep learning models on low-level data representations to solve the emotion recognition task. Since emotion datasets often have a limited amount of data, these approaches may suffer from overfitting, and they may learn based on superficial cues. To address these issues, we propose a novel cross-representation speech model, inspired by disentanglement representation learning, to perform emotion recognition on wav2vec 2.0 speech features. We also train a CNN-based model to recognize emotions from text features extracted with Transformer-based models. We further combine the speech-based and text-based results with a score fusion approach. Our method is evaluated on the IEMOCAP dataset in a 4-class classification problem, and it surpasses current works on speech-only, text-only, and multimodal emotion recognition.