mlsanityData to 7 Oct 2026

Grounding Language Models for Visual Entity Recognition

arXiv 2402.18695 ↗First posted 28 Feb 2024cs.CV cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We introduce AutoVER, an Autoregressive model for Visual Entity Recognition. Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation. It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning. Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever. During inference, a list of retrieved candidate answers explicitly guides language generation by removing invalid decoding paths. The proposed method achieves significant improvements across different dataset splits in the recently proposed Oven-Wiki benchmark. Accuracy on the Entity seen split rises from 32.7% to 61.5%. It also demonstrates superior performance on the unseen and query splits by a substantial double-digit margin.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 64 of its 82 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

64 of 82 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  19. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  20. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  21. 2023A Comprehensive Overview of Large Language Modelscited by 639
  22. 2021ImageNet-21K Pretraining for the Massescited by 607
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  24. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  25. 2020Autoregressive Entity Retrievalcited by 333
  26. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  27. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  28. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  29. 2023Generating Images with Multimodal Language Modelscited by 259
  30. 2022Visual Prompting via Image Inpaintingcited by 248
  31. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  32. 2015Exploring Models and Data for Image Question Answeringcited by 139
  33. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  34. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  35. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  36. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  37. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  38. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 82

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 3; 2026: 2.

12 papers.

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  2. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  3. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  4. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  5. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  6. 2025Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learningcited by 4
  7. 2024Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistantcited by 3
  8. 2026WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognitioncited by 1
  9. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  10. 2024Fine-Grained Open-Vocabulary Object Recognition via User-Guided Segmentationcited by 0
  11. 2024Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linkingcited by 0
  12. 2026Native Association: Confidence-Aware Human Perception in the Wild with a Foundation VLMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models
  2. 2026Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification
  3. 2023MIVC: Multiple Instance Visual Component for Visual-Language Models
  4. 2025V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generation
  5. 2024Non-autoregressive Sequence-to-Sequence Vision-Language Models
  6. 2024Multi-modal Auto-regressive Modeling via Visual Words
  7. 2026E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
  8. 2024Rethinking Negative Instances for Generative Named Entity Recognition