mlsanityData to 7 Oct 2026

Multi-attention Recurrent Network for Human Communication Comprehension

arXiv 1802.00923 ↗First posted 3 Feb 2018cs.AI cs.CL cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape human communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art performance on all the datasets.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 27; 2023: 14; 2024: 14; 2025: 12; 2026: 3.

89 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  3. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 55
  4. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  5. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  6. 2022EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognitioncited by 48
  7. 2022Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representationscited by 48
  8. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  9. 2021Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysiscited by 41
  10. 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 39
  11. 2023A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learningcited by 32
  12. 2021M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysiscited by 29
38 more
  1. 2021The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvementscited by 22
  2. 2021Emotion Recognition from Multiple Modalities: Fundamentals and Methodologiescited by 22
  3. 2022Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognitioncited by 22
  4. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  5. 2022Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysiscited by 20
  6. 2022Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysiscited by 18
  7. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  8. 2021Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactionscited by 16
  9. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  10. 2025Improving Multimodal Learning via Imbalanced Learningcited by 16
  11. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  12. 2025Boosting Multimodal Learning via Disentangled Gradient Learningcited by 15
  13. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  14. 2023CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognitioncited by 12
  15. 2021Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Datacited by 10
  16. 2021Few-Shot Emotion Recognition in Conversation with Sequential Prototypical Networkscited by 10
  17. 2021Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysiscited by 9
  18. 2021CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representationscited by 9
  19. 2022Hybrid Multimodal Fusion for Humor Detectioncited by 9
  20. 2021TEASEL: A Transformer-Based Speech-Prefixed Language Modelcited by 8
  21. 2024Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Spacecited by 8
  22. 2021MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognitioncited by 7
  23. 2026Unbiased Dynamic Multimodal Fusioncited by 7
  24. 2021Graph Capsule Aggregation for Unaligned Multimodal Sequencescited by 6
  25. 2022MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysiscited by 6
  26. 2022Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysiscited by 6
  27. 2022A study on cross-corpus speech emotion recognition and data augmentationcited by 5
  28. 2022AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Workcited by 5
  29. 2023Versatile audio-visual learning for emotion recognitioncited by 5
  30. 2025Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactionscited by 5
  31. 2021Unsupervised Multimodal Language Representations using Convolutional Autoencoderscited by 4
  32. 2022Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Datacited by 4
  33. 2022Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysiscited by 4
  34. 2022Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learningcited by 4
  35. 2023Deep Equilibrium Multimodal Fusioncited by 4
  36. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  37. 2024CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chartcited by 4
  38. 2025Can MLLMs Read the Room? A Multimodal Benchmark for Verifying Truthfulness in Multi-Party Social Interactionscited by 4

50 most cited of 89 All 89 (JSON)