# Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

arXiv: https://arxiv.org/abs/2505.07895 · HTML: https://mlsanity.com/paper/2505.07895/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Simple and Efficient Heterogeneous Graph Neural Network](https://mlsanity.com/paper/2207.02547/) (2022)
- [Embedding Multimodal Relational Data for Knowledge Base Completion](https://mlsanity.com/paper/1809.01341/) (2018)
- [Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering](https://mlsanity.com/paper/2210.10462/) (2022)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [Heterogeneous Graph Learning for Multi-modal Medical Data Analysis](https://mlsanity.com/paper/2211.15158/) (2022)
- [Spectral Heterogeneous Graph Convolutions via Positive Noncommutative Polynomials](https://mlsanity.com/paper/2305.19872/) (2023)
- [Causal-Based Supervision of Attention in Graph Neural Network: A Better and Simpler Choice towards Powerful Attention](https://mlsanity.com/paper/2305.13115/) (2023)
- [Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning](https://mlsanity.com/paper/2310.08032/) (2023)
- [Beyond Graph Convolution: Multimodal Recommendation with Topology-aware MLPs](https://mlsanity.com/paper/2412.11747/) (2024)

## Built on by (most cited first, 1 of 1)

- [Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback](https://mlsanity.com/paper/2609.39456/) (2026)

## Abstract (arXiv)

Nowadays, numerous online platforms can be described as multi-modal heterogeneous networks (MMHNs), such as Douban's movie networks and Amazon's product review networks. Accurately categorizing nodes within these networks is crucial for analyzing the corresponding entities, which requires effective representation learning on nodes. However, existing multi-modal fusion methods often adopt either early fusion strategies which may lose the unique characteristics of individual modalities, or late fusion approaches overlooking the cross-modal guidance in GNN-based information propagation. In this paper, we propose a novel model for node classification in MMHNs, named Heterogeneous Graph Neural Network with Inter-Modal Attention (HGNN-IMA). It learns node representations by capturing the mutual influence of multiple modalities during the information propagation process, within the framework of heterogeneous graph transformer. Specifically, a nested inter-modal attention mechanism is integrated into the inter-node attention to achieve adaptive multi-modal fusion, and modality alignment is also taken into account to encourage the propagation among nodes with consistent similarities across all modalities. Moreover, an attention loss is augmented to mitigate the impact of missing modalities. Extensive experiments validate the superiority of the model in the node classification task, providing an innovative view to handle multi-modal data, especially when accompanied with network structures.