# TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution

arXiv: https://arxiv.org/abs/2603.27723 · HTML: https://mlsanity.com/paper/2603.27723/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings](https://mlsanity.com/paper/2006.13009/) (2020)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs](https://mlsanity.com/paper/2502.00806/) (2025)
- [FedGTA: Topology-aware Averaging for Federated Graph Learning](https://mlsanity.com/paper/2401.11755/) (2024)
- [LGMRec: Local and Global Graph Learning for Multimodal Recommendation](https://mlsanity.com/paper/2312.16400/) (2023)
- [Multimodal Graph Learning for Generative Tasks](https://mlsanity.com/paper/2310.07478/) (2023)
- [FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning](https://mlsanity.com/paper/2404.14061/) (2024)
- [When Graph meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graphs Learning](https://mlsanity.com/paper/2410.09132/) (2024)
- [GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs](https://mlsanity.com/paper/2502.11925/) (2025)
- [Graph4MM: Weaving Multimodal Learning with Structural Information](https://mlsanity.com/paper/2510.16990/) (2025)
- [InstructG2I: Synthesizing Images from Multimodal Attributed Graphs](https://mlsanity.com/paper/2410.07157/) (2024)
- [Towards Effective Federated Graph Foundation Model via Mitigating Knowledge Entanglement](https://mlsanity.com/paper/2505.12684/) (2025)
- [Cross-Contrastive Clustering for Multimodal Attributed Graphs with Dual Graph Filtering](https://mlsanity.com/paper/2511.20030/) (2025)
- [MLaGA: Multimodal Large Language and Graph Assistant](https://mlsanity.com/paper/2506.02568/) (2025)
- [FedBook: A Unified Federated Graph Foundation Codebook with Intra-domain and Inter-domain Knowledge Modeling](https://mlsanity.com/paper/2510.07755/) (2025)
- [A Comprehensive Data-centric Overview of Federated Graph Learning](https://mlsanity.com/paper/2507.16541/) (2025)
- [Rethinking Federated Graph Learning: A Data Condensation Perspective](https://mlsanity.com/paper/2505.02573/) (2025)

## Built on by (most cited first, 3 of 3)

- [RoleMAG: Learning Neighbor Roles in Multimodal Graphs](https://mlsanity.com/paper/2604.12271/) (2026)
- [Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework](https://mlsanity.com/paper/2607.15687/) (2026)
- [Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach](https://mlsanity.com/paper/2610.02881/) (2026)

## Abstract (arXiv)

Multimodal-attributed graphs (MAGs) are a fundamental data structure for multimodal graph learning (MGL), enabling both graph-centric and modality-centric tasks. However, our empirical analysis reveals inherent topology quality limitations in real-world MAGs, including noisy interactions, missing connections, and task-agnostic relational structures. A single graph derived from generic relationships is therefore unlikely to be universally optimal for diverse downstream tasks. To address this challenge, we propose Task-aware Modality and Topology co-Evolution (TMTE), a novel MGL framework that jointly and iteratively optimizes graph topology and multimodal representations toward the target task. TMTE is motivated by the bidirectional coupling between modality and topology: multimodal attributes induce relational structures, while graph topology shapes modality representations. Concretely, TMTE casts topology evolution as multi-perspective metric learning over modality embeddings with an anchor-based approximation, and formulates modality evolution as smoothness-regularized fusion with cross-modal alignment, yielding a closed-loop task-aware co-evolution process. Extensive experiments on 9 MAG datasets and 1 non-graph multimodal dataset across 6 graph-centric and modality-centric tasks show that TMTE consistently achieves state-of-the-art performance. Our code is available at https://anonymous.4open.science/r/TMTE-1873.