# Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition arXiv: https://arxiv.org/abs/2407.21536 ยท HTML: https://mlsanity.com/paper/2407.21536/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 59 references that are in the map. - First posted on arXiv 31 Jul 2024. ## Builds on (most cited first, 24 of 24) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019) - [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020) - [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018) - [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021) - [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021) - [Directed Acyclic Graph Network for Conversational Emotion Recognition](https://mlsanity.com/paper/2105.12907/) (2021) - [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022) - [DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations](https://mlsanity.com/paper/2106.01978/) (2021) - [MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2203.02385/) (2022) - [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022) - [M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2206.02187/) (2022) - [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021) - [Contrast and Generation Make BART a Good Dialogue Emotion Recognizer](https://mlsanity.com/paper/2112.11202/) (2021) - [Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2210.08713/) (2022) - [Graph Based Network with Contextualized Representations of Turns in Dialogue](https://mlsanity.com/paper/2109.04008/) (2021) - [GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2207.12261/) (2022) - [Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations](https://mlsanity.com/paper/2306.01505/) (2023) - [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023) - [Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition](https://mlsanity.com/paper/2308.04502/) (2023) - [Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2010.08218/) (2020) - [Mimicking the Thinking Process for Emotion Recognition in Conversation with Prompts and Paraphrasing](https://mlsanity.com/paper/2306.06601/) (2023) - [Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation](https://mlsanity.com/paper/2403.20289/) (2024) ## Built on by (most cited first, 8 of 8) - [Membership Inference Attacks on Large-Scale Models: A Survey](https://mlsanity.com/paper/2503.19338/) (2025) - [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024) - [Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization](https://mlsanity.com/paper/2501.16106/) (2025) - [Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition](https://mlsanity.com/paper/2602.20530/) (2026) - [Bayesian Spectral Emotion Transition Discovery from Multi-Annotator Disagreement](https://mlsanity.com/paper/2606.01906/) (2026) - [Sync-TVA: A Graph-Attention Framework for Multimodal Emotion Recognition with Cross-Modal Fusion](https://mlsanity.com/paper/2507.21395/) (2025) - [DialogGraph-LLM: Graph-Informed LLMs for End-to-End Audio Dialogue Intent Recognition](https://mlsanity.com/paper/2511.11000/) (2025) - [Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation](https://mlsanity.com/paper/2601.14274/) (2026) ## Abstract (arXiv) Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), thus enabling simultaneous execution of MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.