mlsanityData to 7 Oct 2026

Multi-modal Attention for Speech Emotion Recognition

arXiv 2009.04107 ↗First posted 9 Sep 2020eess.AS cs.MM cs.SDcited by 29

Not on the map. Why?

Abstract (arXiv)

Emotion represents an essential aspect of human speech that is manifested in speech prosody. Speech, visual, and textual cues are complementary in human communication. In this paper, we study a hybrid fusion method, referred to as multi-modal attention network (MMAN) to make use of visual and textual cues in speech emotion recognition. We propose a novel multi-modal attention mechanism, cLSTM-MMA, which facilitates the attention across three modalities and selectively fuse the information. cLSTM-MMA is fused with other uni-modal sub-networks in the late fusion. The experiments show that speech emotion recognition benefits significantly from visual and textual cues, and the proposed cLSTM-MMA alone is as competitive as other fusion methods in terms of accuracy, but with a much more compact network structure. The proposed hybrid network MMAN achieves state-of-the-art performance on IEMOCAP database for emotion recognition.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 0; 2024: 4; 2025: 14; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  2. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  3. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 56
  4. 2025LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformercited by 43
  5. 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 43
  6. 2025OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Datacited by 32
  7. 2025RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformerscited by 26
  8. 2025Edit Transfer: Learning Image Editing via Vision In-Context Relationscited by 23
  9. 2021VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Overcited by 13
  10. 2025OmniPSD: Layered PSD Generation with Diffusion Transformercited by 11
  11. 2025Image Editing As Programs with Diffusion Modelscited by 10
  12. 2021Speaker Attentive Speech Emotion Recognitioncited by 9
17 more
  1. 2021End-to-End Speech Emotion Recognition: Challenges of Real-Life Emergency Call Centers Data Recordingscited by 8
  2. 2021MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognitioncited by 7
  3. 2022MMER: Multimodal Multi-task Learning for Speech Emotion Recognitioncited by 7
  4. 2022Speaker Extraction with Co-Speech Gestures Cuecited by 5
  5. 2025From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarioscited by 3
  6. 2022CTA-RNN: Channel and Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings for Speech Emotion Recognitioncited by 2
  7. 2024Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognitioncited by 2
  8. 2025Learning Annotation Consensus for Continuous Emotion Recognitioncited by 2
  9. 2021Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognitioncited by 1
  10. 2022Late Audio-Visual Fusion for In-The-Wild Speaker Diarizationcited by 1
  11. 2024Self context-aware emotion perception on human-robot interactioncited by 1
  12. 2024Context and System Fusion in Post-ASR Emotion Recognition with Large Language Modelscited by 1
  13. 2025Dynamic Stress Detection: A Study of Temporal Progression Modelling of Stress in Speechcited by 1
  14. 2022VCSE: Time-Domain Visual-Contextual Speaker Extraction Networkcited by 0
  15. 2025WordCraft: Interactive Artistic Typography with Attention Awareness and Noise Blendingcited by 0
  16. 2025In-Context Audio Control of Video Diffusion Transformerscited by 0
  17. 2026DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformercited by 0