# Multi-modal Attention for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2009.04107 · HTML: https://mlsanity.com/paper/2009.04107/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2020.

## Built on by (most cited first, 29 of 29)

- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [OminiControl2: Efficient Conditioning for Diffusion Transformers](https://mlsanity.com/paper/2503.08280/) (2025)
- [MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation](https://mlsanity.com/paper/2502.01572/) (2025)
- [LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer](https://mlsanity.com/paper/2502.01105/) (2025)
- [OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Data](https://mlsanity.com/paper/2505.18445/) (2025)
- [RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers](https://mlsanity.com/paper/2506.02528/) (2025)
- [Edit Transfer: Learning Image Editing via Vision In-Context Relations](https://mlsanity.com/paper/2503.13327/) (2025)
- [VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over](https://mlsanity.com/paper/2110.03342/) (2021)
- [OmniPSD: Layered PSD Generation with Diffusion Transformer](https://mlsanity.com/paper/2512.09247/) (2025)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [Speaker Attentive Speech Emotion Recognition](https://mlsanity.com/paper/2104.07288/) (2021)
- [End-to-End Speech Emotion Recognition: Challenges of Real-Life Emergency Call Centers Data Recordings](https://mlsanity.com/paper/2110.14957/) (2021)
- [MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2111.00865/) (2021)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios](https://mlsanity.com/paper/2506.20279/) (2025)
- [CTA-RNN: Channel and Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2203.17023/) (2022)
- [Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition](https://mlsanity.com/paper/2408.09438/) (2024)
- [Learning Annotation Consensus for Continuous Emotion Recognition](https://mlsanity.com/paper/2505.21196/) (2025)
- [Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition](https://mlsanity.com/paper/2109.07149/) (2021)
- [Late Audio-Visual Fusion for In-The-Wild Speaker Diarization](https://mlsanity.com/paper/2211.01299/) (2022)
- [Self context-aware emotion perception on human-robot interaction](https://mlsanity.com/paper/2401.10946/) (2024)
- [Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models](https://mlsanity.com/paper/2410.03312/) (2024)
- [Dynamic Stress Detection: A Study of Temporal Progression Modelling of Stress in Speech](https://mlsanity.com/paper/2510.08586/) (2025)
- [VCSE: Time-Domain Visual-Contextual Speaker Extraction Network](https://mlsanity.com/paper/2210.06177/) (2022)
- [WordCraft: Interactive Artistic Typography with Attention Awareness and Noise Blending](https://mlsanity.com/paper/2507.09573/) (2025)
- [In-Context Audio Control of Video Diffusion Transformers](https://mlsanity.com/paper/2512.18772/) (2025)
- [DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer](https://mlsanity.com/paper/2605.15681/) (2026)

## Abstract (arXiv)

Emotion represents an essential aspect of human speech that is manifested in speech prosody. Speech, visual, and textual cues are complementary in human communication. In this paper, we study a hybrid fusion method, referred to as multi-modal attention network (MMAN) to make use of visual and textual cues in speech emotion recognition. We propose a novel multi-modal attention mechanism, cLSTM-MMA, which facilitates the attention across three modalities and selectively fuse the information. cLSTM-MMA is fused with other uni-modal sub-networks in the late fusion. The experiments show that speech emotion recognition benefits significantly from visual and textual cues, and the proposed cLSTM-MMA alone is as competitive as other fusion methods in terms of accuracy, but with a much more compact network structure. The proposed hybrid network MMAN achieves state-of-the-art performance on IEMOCAP database for emotion recognition.