# DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognition

arXiv: https://arxiv.org/abs/2312.10579 · HTML: https://mlsanity.com/paper/2312.10579/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning](https://mlsanity.com/paper/2105.09111/) (2021)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2312.06337/) (2023)
- [CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification](https://mlsanity.com/paper/2306.04979/) (2023)
- [CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation](https://mlsanity.com/paper/2108.11626/) (2021)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [COGMEN: COntextualized GNN based Multimodal Emotion recognitioN](https://mlsanity.com/paper/2205.02455/) (2022)
- [Graph Information Bottleneck for Remote Sensing Segmentation](https://mlsanity.com/paper/2312.02545/) (2023)
- [Speech Emotion Recognition with Dual-Sequence LSTM Architecture](https://mlsanity.com/paper/1910.08874/) (2019)
- [CILF-CIAE: CLIP-driven Image-Language Fusion for Correcting Inverse Age Estimation](https://mlsanity.com/paper/2312.01758/) (2023)
- [Eigen-GNN: A Graph Structure Preserving Plug-in for GNNs](https://mlsanity.com/paper/2006.04330/) (2020)
- [Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition](https://mlsanity.com/paper/1811.06937/) (2018)

## Built on by (most cited first, 23 of 23)

- [Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations](https://mlsanity.com/paper/2407.00119/) (2024)
- [Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck](https://mlsanity.com/paper/2408.00295/) (2024)
- [SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samples](https://mlsanity.com/paper/2410.10365/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2312.16778/) (2023)
- [MCSFF: Multi-modal Consistency and Specificity Fusion Framework for Entity Alignment](https://mlsanity.com/paper/2410.14584/) (2024)
- [Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification](https://mlsanity.com/paper/2410.18130/) (2024)
- [A Multi-view Mask Contrastive Learning Graph Convolutional Neural Network for Age Estimation](https://mlsanity.com/paper/2407.16234/) (2024)
- [DuSEGO: Dual Second-order Equivariant Graph Ordinary Differential Equation](https://mlsanity.com/paper/2411.10000/) (2024)
- [Contrastive Multi-graph Learning with Neighbor Hierarchical Sifting for Semi-supervised Text Classification](https://mlsanity.com/paper/2411.16787/) (2024)
- [SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignment](https://mlsanity.com/paper/2410.20733/) (2024)
- [Dynamic Graph Neural ODE Network for Multi-modal Emotion Recognition in Conversation](https://mlsanity.com/paper/2412.02935/) (2024)
- [GroupFace: Imbalanced Age Estimation Based on Multi-hop Attention Graph Convolutional Network and Group-aware Margin Optimization](https://mlsanity.com/paper/2412.11450/) (2024)
- [Scalable Attribute-Missing Graph Clustering via Neighborhood Differentiation](https://mlsanity.com/paper/2507.13368/) (2025)
- [Entity-Aware Self-Attention and Contextualized GCN for Enhanced Relation Extraction in Long Sentences](https://mlsanity.com/paper/2409.13755/) (2024)
- [SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representation](https://mlsanity.com/paper/2412.11652/) (2024)
- [LRA-GNN: Latent Relation-Aware Graph Neural Network with Initial and Dynamic Residual for Facial Age Estimation](https://mlsanity.com/paper/2502.05423/) (2025)
- [An Effective Index for Truss-based Community Search on Large Directed Graphs](https://mlsanity.com/paper/2401.10641/) (2024)
- [SE-GNN: Seed Expanded-Aware Graph Neural Network with Iterative Optimization for Semi-supervised Entity Alignment](https://mlsanity.com/paper/2503.20801/) (2025)
- [BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2503.23990/) (2025)
- [Divide-Then-Rule: A Cluster-Driven Hierarchical Interpolator for Attribute-Missing Graphs](https://mlsanity.com/paper/2507.10595/) (2025)
- [VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2605.18547/) (2026)

## Abstract (arXiv)

With the continuous development of deep learning (DL), the task of multimodal dialogue emotion recognition (MDER) has recently received extensive research attention, which is also an essential branch of DL. The MDER aims to identify the emotional information contained in different modalities, e.g., text, video, and audio, in different dialogue scenes. However, existing research has focused on modeling contextual semantic information and dialogue relations between speakers while ignoring the impact of event relations on emotion. To tackle the above issues, we propose a novel Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition (DER-GCN) method. It models dialogue relations between speakers and captures latent event relations information. Specifically, we construct a weighted multi-relationship graph to simultaneously capture the dependencies between speakers and event relations in a dialogue. Moreover, we also introduce a Self-Supervised Masked Graph Autoencoder (SMGAE) to improve the fusion representation ability of features and structures. Next, we design a new Multiple Information Transformer (MIT) to capture the correlation between different relations, which can provide a better fuse of the multivariate information between relations. Finally, we propose a loss optimization strategy based on contrastive learning to enhance the representation learning ability of minority class features. We conduct extensive experiments on the IEMOCAP and MELD benchmark datasets, which verify the effectiveness of the DER-GCN model. The results demonstrate that our model significantly improves both the average accuracy and the f1 value of emotion recognition.