# Multi-Grained Multimodal Interaction Network for Entity Linking

arXiv: https://arxiv.org/abs/2307.09721 · HTML: https://mlsanity.com/paper/2307.09721/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](https://mlsanity.com/paper/2111.02387/) (2021)
- [Improving Entity Linking by Modeling Latent Relations between Mentions](https://mlsanity.com/paper/1804.10637/) (2018)
- [Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation](https://mlsanity.com/paper/1601.01343/) (2016)
- [Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader](https://mlsanity.com/paper/1905.07098/) (2019)
- [WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types](https://mlsanity.com/paper/2204.06347/) (2022)
- [Multimodal Entity Linking for Tweets](https://mlsanity.com/paper/2104.03236/) (2021)
- [Joint Entity Linking with Deep Reinforcement Learning](https://mlsanity.com/paper/1902.00330/) (2019)
- [Named Entity Disambiguation for Noisy Text](https://mlsanity.com/paper/1706.09147/) (2017)
- [Neural Collective Entity Linking](https://mlsanity.com/paper/1811.08603/) (2018)
- [Learning Dynamic Context Augmentation for Global Entity Linking](https://mlsanity.com/paper/1909.02117/) (2019)
- [Capturing Semantic Similarity for Entity Linking with Convolutional Neural Networks](https://mlsanity.com/paper/1604.00734/) (2016)
- [Entity-aware Transformers for Entity Search](https://mlsanity.com/paper/2205.00820/) (2022)
- [ConCET: Entity-Aware Topic Classification for Open-Domain Conversational Agents](https://mlsanity.com/paper/2005.13798/) (2020)

## Built on by (most cited first, 13 of 13)

- [DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking](https://mlsanity.com/paper/2310.05589/) (2023)
- [Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking](https://mlsanity.com/paper/2406.01934/) (2024)
- [UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models](https://mlsanity.com/paper/2407.16160/) (2024)
- [RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems](https://mlsanity.com/paper/2503.04252/) (2025)
- [KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking](https://mlsanity.com/paper/2504.15135/) (2025)
- [Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation](https://mlsanity.com/paper/2501.14166/) (2025)
- [DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2508.15876/) (2025)
- [OVEL: Large Language Model as Memory Manager for Online Video Entity Linking](https://mlsanity.com/paper/2403.01411/) (2024)
- [VP-MEL: Visual Prompts Guided Multimodal Entity Linking](https://mlsanity.com/paper/2412.06720/) (2024)
- [Multi-level Mixture of Experts for Multimodal Entity Linking](https://mlsanity.com/paper/2507.07108/) (2025)
- [I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking](https://mlsanity.com/paper/2508.02243/) (2025)
- [PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linking](https://mlsanity.com/paper/2510.02726/) (2025)
- [KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs](https://mlsanity.com/paper/2610.16502/) (2026)

## Abstract (arXiv)

Multimodal entity linking (MEL) task, which aims at resolving ambiguous mentions to a multimodal knowledge graph, has attracted wide attention in recent years. Though large efforts have been made to explore the complementary effect among multiple modalities, however, they may fail to fully absorb the comprehensive expression of abbreviated textual context and implicit visual indication. Even worse, the inevitable noisy data may cause inconsistency of different modalities during the learning process, which severely degenerates the performance. To address the above issues, in this paper, we propose a novel Multi-GraIned Multimodal InteraCtion Network (MIMIC) framework for solving the MEL task. Specifically, the unified inputs of mentions and entities are first encoded by textual/visual encoders separately, to extract global descriptive features and local detailed features. Then, to derive the similarity matching score for each mention-entity pair, we device three interaction units to comprehensively explore the intra-modal interaction and inter-modal fusion among features of entities and mentions. In particular, three modules, namely the Text-based Global-Local interaction Unit (TGLU), Vision-based DuaL interaction Unit (VDLU) and Cross-Modal Fusion-based interaction Unit (CMFU) are designed to capture and integrate the fine-grained representation lying in abbreviated text and implicit visual cues. Afterwards, we introduce a unit-consistency objective function via contrastive learning to avoid inconsistency and model degradation. Experimental results on three public benchmark datasets demonstrate that our solution outperforms various state-of-the-art baselines, and ablation studies verify the effectiveness of designed modules.