# COGMEN: COntextualized GNN based Multimodal Emotion recognitioN

arXiv: https://arxiv.org/abs/2205.02455 · HTML: https://mlsanity.com/paper/2205.02455/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Directed Acyclic Graph Network for Conversational Emotion Recognition](https://mlsanity.com/paper/2105.12907/) (2021)
- [DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition](https://mlsanity.com/paper/2012.08695/) (2020)
- [DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations](https://mlsanity.com/paper/2106.01978/) (2021)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Affect-Driven Dialog Generation](https://mlsanity.com/paper/1904.02793/) (2019)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Adapting a Language Model for Controlled Affective Text Generation](https://mlsanity.com/paper/2011.04000/) (2020)
- [Gated Mechanism for Attention Based Multimodal Sentiment Analysis](https://mlsanity.com/paper/2003.01043/) (2020)
- [Fine-Grained Emotion Prediction by Modeling Emotion Definitions](https://mlsanity.com/paper/2107.12135/) (2021)
- [Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts](https://mlsanity.com/paper/2112.01938/) (2021)
- [Domain Authoring Assistant for Intelligent Virtual Agents](https://mlsanity.com/paper/1904.03266/) (2019)
- [Developing emotion recognition for video conference software to support people with autism](https://mlsanity.com/paper/2101.10785/) (2021)

## Built on by (most cited first, 29 of 29)

- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognition](https://mlsanity.com/paper/2312.10579/) (2023)
- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition](https://mlsanity.com/paper/2307.15432/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges](https://mlsanity.com/paper/2407.16804/) (2024)
- [Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2310.04456/) (2023)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [End-to-end Semantic-centric Video-based Multimodal Affective Computing](https://mlsanity.com/paper/2408.07694/) (2024)
- [Multi-Task Learning Framework for Extracting Emotion Cause Span and Entailment in Conversations](https://mlsanity.com/paper/2211.03742/) (2022)
- [DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis](https://mlsanity.com/paper/2504.11082/) (2025)
- [Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering](https://mlsanity.com/paper/2309.04734/) (2023)
- [Towards Data-centric Machine Learning on Directed Graphs: a Survey](https://mlsanity.com/paper/2412.01849/) (2024)
- [Effective Context Modeling Framework for Emotion Recognition in Conversations](https://mlsanity.com/paper/2412.16444/) (2024)
- [HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer](https://mlsanity.com/paper/2305.12369/) (2023)
- [A Contextualized Real-Time Multimodal Emotion Recognition for Conversational Agents using Graph Convolutional Networks in Reinforcement Learning](https://mlsanity.com/paper/2310.18363/) (2023)
- [PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2312.12334/) (2023)
- [Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition](https://mlsanity.com/paper/2401.03000/) (2024)
- [Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weighting](https://mlsanity.com/paper/2411.17674/) (2024)
- [WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2412.05558/) (2024)
- [Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition](https://mlsanity.com/paper/2412.13541/) (2024)
- [Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2505.14143/) (2025)
- [Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations](https://mlsanity.com/paper/2510.08606/) (2025)
- [Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation](https://mlsanity.com/paper/2601.14274/) (2026)
- [AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.10043/) (2026)
- [Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition](https://mlsanity.com/paper/2603.25752/) (2026)
- [ML-SAN: Multi-Level Speaker-Adaptive Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2604.25383/) (2026)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)

## Abstract (arXiv)

Emotions are an inherent part of human interactions, and consequently, it is imperative to develop AI systems that understand and recognize human emotions. During a conversation involving various people, a person's emotions are influenced by the other speaker's utterances and their own emotional state over the utterances. In this paper, we propose COntextualized Graph Neural Network based Multimodal Emotion recognitioN (COGMEN) system that leverages local information (i.e., inter/intra dependency between speakers) and global information (context). The proposed model uses Graph Neural Network (GNN) based architecture to model the complex dependencies (local and global information) in a conversation. Our model gives state-of-the-art (SOTA) results on IEMOCAP and MOSEI datasets, and detailed ablation experiments show the importance of modeling information at both levels.