# Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction

arXiv: https://arxiv.org/abs/2311.04507 · HTML: https://mlsanity.com/paper/2311.04507/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Directed Acyclic Graph Network for Conversational Emotion Recognition](https://mlsanity.com/paper/2105.12907/) (2021)
- [DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition](https://mlsanity.com/paper/2012.08695/) (2020)
- [DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations](https://mlsanity.com/paper/2106.01978/) (2021)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Recurrent Space-time Graph Neural Networks](https://mlsanity.com/paper/1904.05582/) (2019)
- [GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation](https://mlsanity.com/paper/2208.00339/) (2022)

## Built on by (most cited first, 18 of 18)

- [Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects](https://mlsanity.com/paper/2505.20511/) (2025)
- [Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion Recognition](https://mlsanity.com/paper/2402.17269/) (2024)
- [End-to-end Semantic-centric Video-based Multimodal Affective Computing](https://mlsanity.com/paper/2408.07694/) (2024)
- [Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion](https://mlsanity.com/paper/2503.23721/) (2025)
- [Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification](https://mlsanity.com/paper/2504.16856/) (2025)
- [Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement](https://mlsanity.com/paper/2511.05946/) (2025)
- [Effective Context Modeling Framework for Emotion Recognition in Conversations](https://mlsanity.com/paper/2412.16444/) (2024)
- [LyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing](https://mlsanity.com/paper/2405.06483/) (2024)
- [MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing](https://mlsanity.com/paper/2602.00811/) (2026)
- [IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings](https://mlsanity.com/paper/2404.04525/) (2024)
- [Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weighting](https://mlsanity.com/paper/2411.17674/) (2024)
- [Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2505.14143/) (2025)
- [Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations](https://mlsanity.com/paper/2510.08606/) (2025)
- [MIAR: Modality Interaction and Alignment Representation Fuison for Multimodal Emotion](https://mlsanity.com/paper/2601.02414/) (2026)
- [Expert-Guided Multimodal Fusion for Unified Emotion and Sentiment Analysis](https://mlsanity.com/paper/2601.07565/) (2026)
- [Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation](https://mlsanity.com/paper/2601.14274/) (2026)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)
- [They Are Not the Same: Direct Causes Are Not Grounded Emotion Explanations](https://mlsanity.com/paper/2605.25208/) (2026)

## Abstract (arXiv)

Emotion recognition is a crucial task for human conversation understanding. It becomes more challenging with the notion of multimodal data, e.g., language, voice, and facial expressions. As a typical solution, the global- and the local context information are exploited to predict the emotional label for every single sentence, i.e., utterance, in the dialogue. Specifically, the global representation could be captured via modeling of cross-modal interactions at the conversation level. The local one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level. Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modality-specific representations. Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), an novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. Extensive experiments demonstrate the effectiveness of CORECT via its state-of-the-art results on the IEMOCAP and CMU-MOSEI datasets for the multimodal ERC task.