# Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

arXiv: https://arxiv.org/abs/2312.06337 · HTML: https://mlsanity.com/paper/2312.06337/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 68 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Gradient Harmonized Single-stage Detector](https://mlsanity.com/paper/1811.05181/) (2018)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition](https://mlsanity.com/paper/2012.08695/) (2020)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022)
- [M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2206.02187/) (2022)
- [CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification](https://mlsanity.com/paper/2306.04979/) (2023)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification](https://mlsanity.com/paper/2008.06914/) (2020)
- [Deep Point Set Resampling via Gradient Fields](https://mlsanity.com/paper/2111.02045/) (2021)
- [Graph Information Bottleneck for Remote Sensing Segmentation](https://mlsanity.com/paper/2312.02545/) (2023)
- [Deep Auto-Encoders with Sequential Learning for Multimodal Dimensional Emotion Recognition](https://mlsanity.com/paper/2004.13236/) (2020)
- [Augmenting Generative Adversarial Networks for Speech Emotion Recognition](https://mlsanity.com/paper/2005.08447/) (2020)

## Built on by (most cited first, 30 of 47)

- [Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion](https://mlsanity.com/paper/2404.17858/) (2024)
- [Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2404.17862/) (2024)
- [DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognition](https://mlsanity.com/paper/2312.10579/) (2023)
- [Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations](https://mlsanity.com/paper/2407.00119/) (2024)
- [Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck](https://mlsanity.com/paper/2408.00295/) (2024)
- [SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samples](https://mlsanity.com/paper/2410.10365/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2503.06456/) (2025)
- [SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition](https://mlsanity.com/paper/2411.19822/) (2024)
- [Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2312.16778/) (2023)
- [A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning](https://mlsanity.com/paper/2401.01495/) (2024)
- [MCSFF: Multi-modal Consistency and Specificity Fusion Framework for Entity Alignment](https://mlsanity.com/paper/2410.14584/) (2024)
- [Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification](https://mlsanity.com/paper/2410.18130/) (2024)
- [A Multi-view Mask Contrastive Learning Graph Convolutional Neural Network for Age Estimation](https://mlsanity.com/paper/2407.16234/) (2024)
- [DuSEGO: Dual Second-order Equivariant Graph Ordinary Differential Equation](https://mlsanity.com/paper/2411.10000/) (2024)
- [The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection](https://mlsanity.com/paper/2601.15316/) (2026)
- [Contrastive Multi-graph Learning with Neighbor Hierarchical Sifting for Semi-supervised Text Classification](https://mlsanity.com/paper/2411.16787/) (2024)
- [SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignment](https://mlsanity.com/paper/2410.20733/) (2024)
- [Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2409.05243/) (2024)
- [Dynamic Graph Neural ODE Network for Multi-modal Emotion Recognition in Conversation](https://mlsanity.com/paper/2412.02935/) (2024)
- [GroupFace: Imbalanced Age Estimation Based on Multi-hop Attention Graph Convolutional Network and Group-aware Margin Optimization](https://mlsanity.com/paper/2412.11450/) (2024)
- [Scalable Attribute-Missing Graph Clustering via Neighborhood Differentiation](https://mlsanity.com/paper/2507.13368/) (2025)
- [Long-Short Distance Graph Neural Networks and Improved Curriculum Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2507.15205/) (2025)
- [Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning](https://mlsanity.com/paper/2511.07061/) (2025)
- [Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition](https://mlsanity.com/paper/2603.26840/) (2026)
- [Entity-Aware Self-Attention and Contextualized GCN for Enhanced Relation Extraction in Long Sentences](https://mlsanity.com/paper/2409.13755/) (2024)
- [SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representation](https://mlsanity.com/paper/2412.11652/) (2024)
- [LRA-GNN: Latent Relation-Aware Graph Neural Network with Initial and Dynamic Residual for Facial Age Estimation](https://mlsanity.com/paper/2502.05423/) (2025)
- [HIPPD: Brain-Inspired Hierarchical Information Processing for Personality Detection](https://mlsanity.com/paper/2510.09893/) (2025)

## Abstract (arXiv)

The main task of Multimodal Emotion Recognition in Conversations (MERC) is to identify the emotions in modalities, e.g., text, audio, image and video, which is a significant development direction for realizing machine intelligence. However, many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. To tackle this problem, we systematically analyze it from three aspects: data augmentation, loss sensitivity, and sampling strategy, and propose the Class Boundary Enhanced Representation Learning (CBERL) model. Concretely, we first design a multimodal generative adversarial network to address the imbalanced distribution of {emotion} categories in raw data. Secondly, a deep joint variational autoencoder is proposed to fuse complementary semantic information across modalities and obtain discriminative feature representations. Finally, we implement a multi-task graph neural network with mask reconstruction and classification optimization to solve the problem of overfitting and underfitting in class boundary learning, and achieve cross-modal emotion recognition. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition. Especially on the minority class fear and disgust emotion labels, our model improves the accuracy and F1 value by 10% to 20%.