# Multimodal Speech Emotion Recognition Using Audio and Text

arXiv: https://arxiv.org/abs/1810.04635 · HTML: https://mlsanity.com/paper/1810.04635/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2018.

## Built on by (most cited first, 30 of 55)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [Learning Emotion Representations from Verbal and Nonverbal Communication](https://mlsanity.com/paper/2305.13500/) (2023)
- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [Towards Relatable Explainable AI with the Perceptual Process](https://mlsanity.com/paper/2112.14005/) (2021)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?](https://mlsanity.com/paper/2202.09263/) (2022)
- [Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques](https://mlsanity.com/paper/2406.08353/) (2024)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Speaker Normalization for Self-supervised Speech Emotion Recognition](https://mlsanity.com/paper/2202.01252/) (2022)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation](https://mlsanity.com/paper/2108.02510/) (2021)
- [VEATIC: Video-based Emotion and Affect Tracking in Context Dataset](https://mlsanity.com/paper/2309.06745/) (2023)
- [HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2304.06910/) (2023)
- [Bias and Fairness on Multimodal Emotion Detection Algorithms](https://mlsanity.com/paper/2205.08383/) (2022)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [Multistage linguistic conditioning of convolutional layers for speech emotion recognition](https://mlsanity.com/paper/2110.06650/) (2021)
- [Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition](https://mlsanity.com/paper/2201.11826/) (2022)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [Team Phoenix at WASSA 2021: Emotion Analysis on News Stories with Pre-Trained Language Models](https://mlsanity.com/paper/2103.06057/) (2021)
- [Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data](https://mlsanity.com/paper/2210.17311/) (2022)
- [A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion](https://mlsanity.com/paper/2307.11584/) (2023)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data](https://mlsanity.com/paper/2204.04645/) (2022)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Pretrained audio neural networks for Speech emotion recognition in Portuguese](https://mlsanity.com/paper/2210.14716/) (2022)
- [LEAN: Light and Efficient Audio Classification Network](https://mlsanity.com/paper/2305.12712/) (2023)

## Abstract (arXiv)

Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers. In this paper, we propose a novel deep dual recurrent encoder model that utilizes text data and audio signals simultaneously to obtain a better understanding of speech data. As emotional dialogue is composed of sound and spoken content, our model encodes the information from audio and text sequences using dual recurrent neural networks (RNNs) and then combines the information from these sources to predict the emotion class. This architecture analyzes speech data from the signal level to the language level, and it thus utilizes the information within the data more comprehensively than models that focus on audio features. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods in assigning data to one of four emotion categories (i.e., angry, happy, sad and neutral) when the model is applied to the IEMOCAP dataset, as reflected by accuracies ranging from 68.8% to 71.8%.