mlsanityData to 7 Oct 2026

Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities

arXiv 2302.11154 ↗First posted 22 Feb 2023cs.CV cs.AI cs.CLcited by 112

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit strong generalization on various visual domains and tasks. However, existing image classification benchmarks often evaluate recognition on a specific domain (e.g., outdoor images) or a specific task (e.g., classifying plant species), which falls short of evaluating whether pre-trained foundational models are universal visual recognizers. To address this, we formally present the task of Open-domain Visual Entity recognitioN (OVEN), where a model need to link an image onto a Wikipedia entity with respect to a text query. We construct OVEN-Wiki by re-purposing 14 existing datasets with all labels grounded onto one single label space: Wikipedia entities. OVEN challenges models to select among six million possible Wikipedia entities, making it a general visual recognition benchmark with the largest number of labels. Our study on state-of-the-art pre-trained models reveals large headroom in generalizing to the massive-scale label space. We show that a PaLI-based auto-regressive visual recognition model performs surprisingly well, even on Wikipedia entities that have never been seen during fine-tuning. We also find existing pretrained models yield different strengths: while PaLI-based models obtain higher overall performance, CLIP-based models are better at recognizing tail entities.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

48 of 67 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
36 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2016Matching Networks for One Shot Learningcited by 2,246
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2021Scaling Vision Transformerscited by 993
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2021ImageNet-21K Pretraining for the Massescited by 607
  15. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  16. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  17. 2020Autoregressive Entity Retrievalcited by 333
  18. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  19. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  20. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  21. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  22. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  23. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  24. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  25. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  26. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  27. 2021WebQA: Multihop and Multimodal QAcited by 109
  28. 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
  29. 2022All You May Need for VQA are Image Captionscited by 65
  30. 2020Entity Linking in 100 Languagescited by 52
  31. 2021Multilingual Autoregressive Entity Linkingcited by 50
  32. 2021Zero-shot Visual Question Answering using Knowledge Graphcited by 37
  33. 2021EntQA: Entity Linking as Question Answeringcited by 34
  34. 2017VSE++: Improving Visual-Semantic Embeddings with Hard Negativescited by 20
  35. 2020Two-stage Discriminative Re-ranking for Large-scale Landmark Retrievalcited by 18
  36. 2021Revisiting Document Representations for Large-Scale Zero-Shot Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 27; 2025: 38; 2026: 36.

112 papers.

Sort:

By area:

  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  2. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  3. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  4. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  5. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  6. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  7. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  8. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  9. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  10. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  11. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  12. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
38 more
  1. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  3. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  4. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  5. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  6. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  7. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  8. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  9. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  10. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  11. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  12. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
  13. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
  14. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 19
  15. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  16. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  17. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  18. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  19. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  20. 2025Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligencecited by 15
  21. 2025COVE: COntext and VEracity prediction for out-of-context imagescited by 14
  22. 2025Towards General Continuous Memory for Vision-Language Modelscited by 14
  23. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  24. 2024Grounding Language Models for Visual Entity Recognitioncited by 12
  25. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  26. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  27. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  28. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  29. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  30. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  31. 2025Closing the Modality Gap for Mixed Modality Searchcited by 11
  32. 2024MATE: Meet At The Embedding -- Connecting Images with Long Textscited by 10
  33. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  34. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  35. 2023Fully Authentic Visual Question Answering Dataset from Online Communitiescited by 9
  36. 2024A Generative Approach for Wikipedia-Scale Visual Entity Recognitioncited by 9
  37. 2023AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributescited by 8
  38. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8

50 most cited of 112 All 112 (JSON)