# WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

arXiv: https://arxiv.org/abs/2603.09921 · HTML: https://mlsanity.com/paper/2603.09921/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](https://mlsanity.com/paper/2302.11713/) (2023)
- [SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models](https://mlsanity.com/paper/2402.05935/) (2024)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities](https://mlsanity.com/paper/2302.11154/) (2023)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2305.07011/) (2023)
- [Good News, Everyone! Context driven entity-aware captioning for news images](https://mlsanity.com/paper/1904.01475/) (2019)
- [EchoSight: Advancing Visual-Language Models with Wiki Knowledge](https://mlsanity.com/paper/2407.12735/) (2024)
- [HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models](https://mlsanity.com/paper/2303.15786/) (2023)

## Built on by (most cited first, 1 of 1)

- [Beyond Readability: Evaluating Task Information Recoverability](https://mlsanity.com/paper/2609.36957/) (2026)

## Abstract (arXiv)

Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large language model embeddings as knowledge-rich entity representations and enhances them with a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/projectpages/WikiCLIP/