# Multi-level Matching Network for Multimodal Entity Linking

arXiv: https://arxiv.org/abs/2412.10440 · HTML: https://mlsanity.com/paper/2412.10440/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](https://mlsanity.com/paper/2111.02387/) (2021)
- [Deep Joint Entity Disambiguation with Local Neural Attention](https://mlsanity.com/paper/1704.04920/) (2017)
- [Robust Cross-Modal Representation Learning with Progressive Self-Distillation](https://mlsanity.com/paper/2204.04588/) (2022)
- [Improving Entity Linking by Modeling Latent Relations between Mentions](https://mlsanity.com/paper/1804.10637/) (2018)
- [Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader](https://mlsanity.com/paper/1905.07098/) (2019)
- [WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types](https://mlsanity.com/paper/2204.06347/) (2022)
- [MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid](https://mlsanity.com/paper/2212.14454/) (2022)
- [Multimodal Entity Linking for Tweets](https://mlsanity.com/paper/2104.03236/) (2021)
- [Joint Entity Linking with Deep Reinforcement Learning](https://mlsanity.com/paper/1902.00330/) (2019)
- [Dynamically Expandable Graph Convolution for Streaming Recommendation](https://mlsanity.com/paper/2303.11700/) (2023)
- [Neural Collective Entity Linking](https://mlsanity.com/paper/1811.08603/) (2018)
- [Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment](https://mlsanity.com/paper/2307.16210/) (2023)
- [Atomic Fact Decomposition Helps Attributed Question Answering](https://mlsanity.com/paper/2410.16708/) (2024)
- [Learning Dynamic Context Augmentation for Global Entity Linking](https://mlsanity.com/paper/1909.02117/) (2019)
- [An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering](https://mlsanity.com/paper/2303.10368/) (2023)
- [Context-aware Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2109.07990/) (2021)
- [DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking](https://mlsanity.com/paper/2310.05589/) (2023)
- [ConCET: Entity-Aware Topic Classification for Open-Domain Conversational Agents](https://mlsanity.com/paper/2005.13798/) (2020)
- [Leveraging Intra-modal and Inter-modal Interaction for Multi-Modal Entity Alignment](https://mlsanity.com/paper/2404.17590/) (2024)

## Built on by (most cited first, 2 of 2)

- [KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking](https://mlsanity.com/paper/2504.15135/) (2025)
- [Multi-level Mixture of Experts for Multimodal Entity Linking](https://mlsanity.com/paper/2507.07108/) (2025)

## Abstract (arXiv)

Multimodal entity linking (MEL) aims to link ambiguous mentions within multimodal contexts to corresponding entities in a multimodal knowledge base. Most existing approaches to MEL are based on representation learning or vision-and-language pre-training mechanisms for exploring the complementary effect among multiple modalities. However, these methods suffer from two limitations. On the one hand, they overlook the possibility of considering negative samples from the same modality. On the other hand, they lack mechanisms to capture bidirectional cross-modal interaction. To address these issues, we propose a Multi-level Matching network for Multimodal Entity Linking (M3EL). Specifically, M3EL is composed of three different modules: (i) a Multimodal Feature Extraction module, which extracts modality-specific representations with a multimodal encoder and introduces an intra-modal contrastive learning sub-module to obtain better discriminative embeddings based on uni-modal differences; (ii) an Intra-modal Matching Network module, which contains two levels of matching granularity: Coarse-grained Global-to-Global and Fine-grained Global-to-Local, to achieve local and global level intra-modal interaction; (iii) a Cross-modal Matching Network module, which applies bidirectional strategies, Textual-to-Visual and Visual-to-Textual matching, to implement bidirectional cross-modal interaction. Extensive experiments conducted on WikiMEL, RichpediaMEL, and WikiDiverse datasets demonstrate the outstanding performance of M3EL when compared to the state-of-the-art baselines.