# Visual Named Entity Linking: A New Dataset and A Baseline

arXiv: https://arxiv.org/abs/2211.04872 · HTML: https://mlsanity.com/paper/2211.04872/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Zero-Shot Entity Linking by Reading Entity Descriptions](https://mlsanity.com/paper/1906.07348/) (2019)
- [Good News, Everyone! Context driven entity-aware captioning for news images](https://mlsanity.com/paper/1904.01475/) (2019)
- [Transform and Tell: Entity-Aware News Image Captioning](https://mlsanity.com/paper/2004.08070/) (2020)
- [Neural Entity Linking: A Survey of Models Based on Deep Learning](https://mlsanity.com/paper/2006.00575/) (2020)
- [WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types](https://mlsanity.com/paper/2204.06347/) (2022)
- [Multimodal Entity Linking for Tweets](https://mlsanity.com/paper/2104.03236/) (2021)
- [Evaluating the Impact of Knowledge Graph Context on Entity Disambiguation Models](https://mlsanity.com/paper/2008.05190/) (2020)
- [BreakingNews: Article Annotation by Image and Text Processing](https://mlsanity.com/paper/1603.07141/) (2016)
- [Neural Cross-Lingual Entity Linking](https://mlsanity.com/paper/1712.01813/) (2017)

## Built on by (most cited first, 15 of 15)

- [Retrieval-based Knowledge Augmented Vision Language Pre-training](https://mlsanity.com/paper/2304.13923/) (2023)
- [MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entities](https://mlsanity.com/paper/2307.14878/) (2023)
- [Cross-modal Retrieval for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2401.05736/) (2024)
- [A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking](https://mlsanity.com/paper/2312.11816/) (2023)
- [AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes](https://mlsanity.com/paper/2305.14725/) (2023)
- [Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant](https://mlsanity.com/paper/2410.19144/) (2024)
- [DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2508.15876/) (2025)
- [DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2404.04818/) (2024)
- [OVEL: Large Language Model as Memory Manager for Online Video Entity Linking](https://mlsanity.com/paper/2403.01411/) (2024)
- [DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model](https://mlsanity.com/paper/2407.12019/) (2024)
- [CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation](https://mlsanity.com/paper/2506.09109/) (2025)
- [KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge](https://mlsanity.com/paper/2508.14080/) (2025)
- [Illuminating Visual Identity in Universal Multimodal Embeddings](https://mlsanity.com/paper/2608.01794/) (2026)
- [Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences](https://mlsanity.com/paper/2404.08666/) (2024)
- [M3EL: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking](https://mlsanity.com/paper/2410.18096/) (2024)

## Abstract (arXiv)

Visual Entity Linking (VEL) is a task to link regions of images with their corresponding entities in Knowledge Bases (KBs), which is beneficial for many computer vision tasks such as image retrieval, image caption, and visual question answering. While existing tasks in VEL either rely on textual data to complement a multi-modal linking or only link objects with general entities, which fails to perform named entity linking on large amounts of image data. In this paper, we consider a purely Visual-based Named Entity Linking (VNEL) task, where the input only consists of an image. The task is to identify objects of interest (i.e., visual entity mentions) in images and link them to corresponding named entities in KBs. Since each entity often contains rich visual and textual information in KBs, we thus propose three different sub-tasks, i.e., visual to visual entity linking (V2VEL), visual to textual entity linking (V2TEL), and visual to visual-textual entity linking (V2VTEL). In addition, we present a high-quality human-annotated visual person linking dataset, named WIKIPerson. Based on WIKIPerson, we establish a series of baseline algorithms for the solution of each sub-task, and conduct experiments to verify the quality of proposed datasets and the effectiveness of baseline methods. We envision this work to be helpful for soliciting more works regarding VNEL in the future. The codes and datasets are publicly available at https://github.com/ict-bigdatalab/VNEL.