# Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

arXiv: https://arxiv.org/abs/2405.17900 · HTML: https://mlsanity.com/paper/2405.17900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Efficient Multimodal Fusion via Interactive Prompting](https://mlsanity.com/paper/2304.06306/) (2023)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition](https://mlsanity.com/paper/2312.10201/) (2023)
- [Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.09856/) (2023)
- [QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis](https://mlsanity.com/paper/2303.07682/) (2023)
- [Mimicking the Thinking Process for Emotion Recognition in Conversation with Prompts and Paraphrasing](https://mlsanity.com/paper/2306.06601/) (2023)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Improving EEG-based Emotion Recognition by Fusing Time-frequency And Spatial Representations](https://mlsanity.com/paper/2303.11421/) (2023)

## Abstract (arXiv)

The purpose of emotion recognition in conversation (ERC) is to identify the emotion category of an utterance based on contextual information. Previous ERC methods relied on simple connections for cross-modal fusion and ignored the information differences between modalities, resulting in the model being unable to focus on modality-specific emotional information. At the same time, the shared information between modalities was not processed to generate emotions. Information redundancy problem. To overcome these limitations, we propose a cross-modal fusion emotion prediction network based on vector connections. The network mainly includes two stages: the multi-modal feature fusion stage based on connection vectors and the emotion classification stage based on fused features. Furthermore, we design a supervised inter-class contrastive learning module based on emotion labels. Experimental results confirm the effectiveness of the proposed method, demonstrating excellent performance on the IEMOCAP and MELD datasets.