# IMF: Interactive Multimodal Fusion Model for Link Prediction

arXiv: https://arxiv.org/abs/2303.10816 · HTML: https://mlsanity.com/paper/2303.10816/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network](https://mlsanity.com/paper/1712.02121/) (2017)
- [RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems](https://mlsanity.com/paper/1803.03467/) (2018)
- [Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction](https://mlsanity.com/paper/1911.09419/) (2019)
- [Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs](https://mlsanity.com/paper/1906.01195/) (2019)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [A Re-evaluation of Knowledge Graph Completion Methods](https://mlsanity.com/paper/1911.03903/) (2019)
- [On Sampling Strategies for Neural Network-based Collaborative Filtering](https://mlsanity.com/paper/1706.07881/) (2017)
- [Image-embodied Knowledge Representation Learning](https://mlsanity.com/paper/1609.07028/) (2016)
- [MUREL: Multimodal Relational Reasoning for Visual Question Answering](https://mlsanity.com/paper/1902.09487/) (2019)
- [InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions](https://mlsanity.com/paper/1911.00219/) (2019)
- [AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations](https://mlsanity.com/paper/2002.11252/) (2020)
- [Embedding Multimodal Relational Data for Knowledge Base Completion](https://mlsanity.com/paper/1809.01341/) (2018)
- [AutoSF: Searching Scoring Functions for Knowledge Graph Embedding](https://mlsanity.com/paper/1904.11682/) (2019)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)

## Built on by (most cited first, 28 of 28)

- [NativE: Multi-modal Knowledge Graph Completion in the Wild](https://mlsanity.com/paper/2406.17605/) (2024)
- [Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation](https://mlsanity.com/paper/2404.09468/) (2024)
- [Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning](https://mlsanity.com/paper/2405.16869/) (2024)
- [Contrasformer: A Brain Network Contrastive Transformer for Neurodegenerative Condition Identification](https://mlsanity.com/paper/2409.10944/) (2024)
- [A Survey on Large Language Models in Multimodal Recommender Systems](https://mlsanity.com/paper/2505.09777/) (2025)
- [Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2412.18733/) (2024)
- [Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs](https://mlsanity.com/paper/2401.00608/) (2023)
- [Zero-Shot Relational Learning for Multimodal Knowledge Graphs](https://mlsanity.com/paper/2404.06220/) (2024)
- [Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2509.06074/) (2025)
- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)
- [When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs](https://mlsanity.com/paper/2604.02778/) (2026)
- [Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2407.02867/) (2024)
- [Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts](https://mlsanity.com/paper/2501.15688/) (2025)
- [Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2506.22036/) (2025)
- [Complementarity-driven Representation Learning for Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2507.20620/) (2025)
- [ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2510.16753/) (2025)
- [HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs](https://mlsanity.com/paper/2506.00826/) (2025)
- [Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completion](https://mlsanity.com/paper/2509.23714/) (2025)
- [A Modality-Aware Cooperative Co-Evolutionary Framework for Multimodal Graph Neural Architecture Search](https://mlsanity.com/paper/2510.07325/) (2025)
- [Every Little Helps: Building Knowledge Graph Foundation Model with Fine-grained Transferable Multi-modal Tokens](https://mlsanity.com/paper/2602.15896/) (2026)
- [E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications](https://mlsanity.com/paper/2602.20877/) (2026)
- [Tokenization, Fusion and Decoupling: Bridging the Granularity Mismatch Between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2602.22698/) (2026)
- [Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment](https://mlsanity.com/paper/2603.16974/) (2026)
- [DyMRL: Dynamic Multispace Representation Learning for Multimodal Event Forecasting in Knowledge Graph](https://mlsanity.com/paper/2603.24636/) (2026)
- [CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs](https://mlsanity.com/paper/2605.10510/) (2026)
- [Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/2607.09777/) (2026)
- [MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2607.15592/) (2026)
- [More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning](https://mlsanity.com/paper/2608.29139/) (2026)

## Abstract (arXiv)

Link prediction aims to identify potential missing triples in knowledge graphs. To get better results, some recent studies have introduced multimodal information to link prediction. However, these methods utilize multimodal information separately and neglect the complicated interaction between different modalities. In this paper, we aim at better modeling the inter-modality information and thus introduce a novel Interactive Multimodal Fusion (IMF) model to integrate knowledge from different modalities. To this end, we propose a two-stage multimodal fusion framework to preserve modality-specific knowledge as well as take advantage of the complementarity between different modalities. Instead of directly projecting different modalities into a unified space, our multimodal fusion module limits the representations of different modalities independent while leverages bilinear pooling for fusion and incorporates contrastive learning as additional constraints. Furthermore, the decision fusion module delivers the learned weighted average over the predictions of all modalities to better incorporate the complementarity of different modalities. Our approach has been demonstrated to be effective through empirical evaluations on several real-world datasets. The implementation code is available online at https://github.com/HestiaSky/IMF-Pytorch.