# Deep Multimodal Learning for Emotion Recognition in Spoken Language

arXiv: https://arxiv.org/abs/1802.08332 · HTML: https://mlsanity.com/paper/1802.08332/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 3 of 3)

- [Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition](https://mlsanity.com/paper/2404.09559/) (2024)
- [DyKen-Hyena: Dynamic Kernel Generation via Cross-Modal Attention for Multimodal Intent Recognition](https://mlsanity.com/paper/2509.09940/) (2025)
- [Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models](https://mlsanity.com/paper/2602.00123/) (2026)

## Abstract (arXiv)

In this paper, we present a novel deep multimodal framework to predict human emotions based on sentence-level spoken language. Our architecture has two distinctive characteristics. First, it extracts the high-level features from both text and audio via a hybrid deep multimodal structure, which considers the spatial information from text, temporal information from audio, and high-level associations from low-level handcrafted features. Second, we fuse all features by using a three-layer deep neural network to learn the correlations across modalities and train the feature extraction and fusion modules together, allowing optimal global fine-tuning of the entire structure. We evaluated the proposed framework on the IEMOCAP dataset. Our result shows promising performance, achieving 60.4% in weighted accuracy for five emotion categories.