mlsanityData to 7 Oct 2026

A Generative Approach for Wikipedia-Scale Visual Entity Recognition

arXiv 2403.02041 ↗First posted 4 Mar 2024cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we address web-scale visual entity recognition, specifically the task of mapping a given query image to one of the 6 million existing entities in Wikipedia. One way of approaching a problem of such scale is using dual-encoder models (eg CLIP), where all the entity names and query images are embedded into a unified space, paving the way for an approximate k-NN search. Alternatively, it is also possible to re-purpose a captioning model to directly generate the entity names for a given image. In contrast, we introduce a novel Generative Entity Recognition (GER) framework, which given an input image learns to auto-regressively decode a semantic and discriminative ``code'' identifying the target entity. Our experiments demonstrate the efficacy of this GER paradigm, showcasing state-of-the-art performance on the challenging OVEN benchmark. GER surpasses strong captioning, dual-encoder, visual matching and hierarchical classification baselines, affirming its advantage in tackling the complexities of web-scale recognition.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

37 of 49 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2021Scaling Vision Transformerscited by 993
25 more
  1. 2019When Does Label Smoothing Help?cited by 985
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  7. 2020Autoregressive Entity Retrievalcited by 333
  8. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  9. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  10. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  11. 2023Recommender Systems with Generative Retrievalcited by 191
  12. 2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
  13. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  14. 2022A Neural Corpus Indexer for Document Retrievalcited by 107
  15. 2023Learning to Tokenize for Generative Retrievalcited by 66
  16. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  17. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  18. 2022Learning with Neighbor Consistency for Noisy Labelscited by 55
  19. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  20. 2022WebFace260M: A Benchmark for Million-Scale Deep Face Recognitioncited by 41
  21. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  22. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  23. 2023Improving Image Recognition by Retrieving from Web-Scale Image-Text Datacited by 20
  24. 2023IRGen: Generative Modeling for Image Retrievalcited by 18
  25. 2023Multi-modal Extreme Classificationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  2. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  3. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  4. 2025Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learningcited by 4
  5. 2024Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistantcited by 3
  6. 2025GIF: Generative Inspiration for Face Recognition at Scalecited by 2
  7. 2026WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognitioncited by 1
  8. 2026Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identificationcited by 1
  9. 2024Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linkingcited by 0