# Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning

arXiv: https://arxiv.org/abs/2508.18322 · HTML: https://mlsanity.com/paper/2508.18322/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 24 Aug 2025.

## Builds on (most cited first, 18 of 18)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [MSAF: Multimodal Split Attention Fusion](https://mlsanity.com/paper/2012.07175/) (2020)
- [Gated Mechanism for Attention Based Multimodal Sentiment Analysis](https://mlsanity.com/paper/2003.01043/) (2020)
- [Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion](https://mlsanity.com/paper/2011.13572/) (2020)
- [Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition](https://mlsanity.com/paper/2010.02057/) (2020)
- [Graph Capsule Aggregation for Unaligned Multimodal Sequences](https://mlsanity.com/paper/2108.07543/) (2021)
- [Robust Multimodal Learning via Cross-Modal Proxy Tokens](https://mlsanity.com/paper/2501.17823/) (2025)
- [Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment](https://mlsanity.com/paper/2503.09498/) (2025)

## Built on by (most cited first, 1 of 1)

- [CAST: Collapse-Aware multi-Scale Topology Fusion for Multimodal Coreset Selection](https://mlsanity.com/paper/2605.11705/) (2026)

## Abstract (arXiv)

Multimodal sentiment analysis (MSA) aims to infer emotional states by effectively integrating textual, acoustic, and visual modalities. Despite notable progress, existing multimodal fusion methods often neglect modality-specific structural dependencies and semantic misalignment, limiting their quality, interpretability, and robustness. To address these challenges, we propose a novel framework called the Structural-Semantic Unifier (SSU), which systematically integrates modality-specific structural information and cross-modal semantic grounding for enhanced multimodal representations. Specifically, SSU dynamically constructs modality-specific graphs by leveraging linguistic syntax for text and a lightweight, text-guided attention mechanism for acoustic and visual modalities, thus capturing detailed intra-modal relationships and semantic interactions. We further introduce a semantic anchor, derived from global textual semantics, that serves as a cross-modal alignment hub, effectively harmonizing heterogeneous semantic spaces across modalities. Additionally, we develop a multiview contrastive learning objective that promotes discriminability, semantic consistency, and structural coherence across intra- and inter-modal views. Extensive evaluations on two widely used benchmark datasets, CMU-MOSI and CMU-MOSEI, demonstrate that SSU consistently achieves state-of-the-art performance while significantly reducing computational overhead compared to prior methods. Comprehensive qualitative analyses further validate SSU's interpretability and its ability to capture nuanced emotional patterns through semantically grounded interactions.