# Multimodal Entity Linking for Tweets

arXiv: https://arxiv.org/abs/2104.03236 · HTML: https://mlsanity.com/paper/2104.03236/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Named Entity Disambiguation for Noisy Text](https://mlsanity.com/paper/1706.09147/) (2017)
- [Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/1808.07793/) (2018)

## Built on by (most cited first, 28 of 28)

- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types](https://mlsanity.com/paper/2204.06347/) (2022)
- [Entity Linking Meets Deep Learning: Techniques and Solutions](https://mlsanity.com/paper/2109.12520/) (2021)
- [Visual Named Entity Linking: A New Dataset and A Baseline](https://mlsanity.com/paper/2211.04872/) (2022)
- [Multi-Grained Multimodal Interaction Network for Entity Linking](https://mlsanity.com/paper/2307.09721/) (2023)
- [Generative Multimodal Entity Linking](https://mlsanity.com/paper/2306.12725/) (2023)
- [A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking](https://mlsanity.com/paper/2312.11816/) (2023)
- [Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia](https://mlsanity.com/paper/2211.00732/) (2022)
- [AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes](https://mlsanity.com/paper/2305.14725/) (2023)
- [DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking](https://mlsanity.com/paper/2310.05589/) (2023)
- [Benchmarking Diverse-Modal Entity Linking with Generative Models](https://mlsanity.com/paper/2305.17337/) (2023)
- [Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking](https://mlsanity.com/paper/2406.01934/) (2024)
- [UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models](https://mlsanity.com/paper/2407.16160/) (2024)
- [KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking](https://mlsanity.com/paper/2504.15135/) (2025)
- [Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections](https://mlsanity.com/paper/2305.14588/) (2023)
- [DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2508.15876/) (2025)
- [Learning Semantic Ambiguities for Zero-Shot Learning](https://mlsanity.com/paper/2201.01823/) (2022)
- [DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2404.04818/) (2024)
- [Multi-level Matching Network for Multimodal Entity Linking](https://mlsanity.com/paper/2412.10440/) (2024)
- [Multimodal Entity Tagging with Multimodal Knowledge Base](https://mlsanity.com/paper/2201.00693/) (2021)
- [OVEL: Large Language Model as Memory Manager for Online Video Entity Linking](https://mlsanity.com/paper/2403.01411/) (2024)
- [DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model](https://mlsanity.com/paper/2407.12019/) (2024)
- [Exploring Large Language Models for Multimodal Sentiment Analysis: Challenges, Benchmarks, and Future Directions](https://mlsanity.com/paper/2411.15408/) (2024)
- [MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking](https://mlsanity.com/paper/2510.14307/) (2025)
- [VP-MEL: Visual Prompts Guided Multimodal Entity Linking](https://mlsanity.com/paper/2412.06720/) (2024)
- [Multi-level Mixture of Experts for Multimodal Entity Linking](https://mlsanity.com/paper/2507.07108/) (2025)
- [I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking](https://mlsanity.com/paper/2508.02243/) (2025)
- [PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linking](https://mlsanity.com/paper/2510.02726/) (2025)

## Abstract (arXiv)

In many information extraction applications, entity linking (EL) has emerged as a crucial task that allows leveraging information about named entities from a knowledge base. In this paper, we address the task of multimodal entity linking (MEL), an emerging research field in which textual and visual information is used to map an ambiguous mention to an entity in a knowledge base (KB). First, we propose a method for building a fully annotated Twitter dataset for MEL, where entities are defined in a Twitter KB. Then, we propose a model for jointly learning a representation of both mentions and entities from their textual and visual contexts. We demonstrate the effectiveness of the proposed model by evaluating it on the proposed dataset and highlight the importance of leveraging visual information when it is available.