mlsanityData to 7 Oct 2026

SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge

arXiv 2405.14554 ↗First posted 23 May 2024cs.CV cs.AIcited by 20

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently due to the large amount of resources required, and therefore fail in many cases. For example, if a LVLM was released on January 2024, and it wouldn't know the singer of the theme song for the new Detective Conan movie, which wasn't released until April 2024. To solve the problem, a promising solution motivated by retrieval-augmented generation (RAG) is to provide LVLMs with up-to-date knowledge via internet search during inference, i.e., internet-augmented generation (IAG), which is already integrated in some closed-source commercial LVLMs such as GPT-4V. However, the specific mechanics underpinning them remain a mystery. In this paper, we propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs. A hierarchical filtering model is trained to effectively and efficiently find the most helpful content from the websites returned by a search engine to prompt LVLMs with up-to-date knowledge. To train the model and evaluate our framework's performance, we propose a pipeline to automatically generate news-related VQA samples to construct a dataset, dubbed UDK-VQA. A multi-model voting mechanism is introduced to label the usefulness of website/content for VQA samples to construct the training set. Experimental results demonstrate the effectiveness of our framework, outperforming GPT-4V by about 25% in accuracy.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 51 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

35 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  2. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
21 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  10. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  13. 2021Internet-Augmented Dialogue Generationcited by 217
  14. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  15. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  16. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  17. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  18. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  19. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  20. 2021Coarse-to-Fine Reasoning for Visual Question Answeringcited by 25
  21. 2023ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Humancited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 8.

20 papers.

Sort:

By area:

  1. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  2. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  3. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  4. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  5. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  6. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  7. 2024FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Modelscited by 6
  8. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 6
  9. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  10. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  11. 2025GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memorycited by 4
  12. 2025IA-T2I: Internet-Augmented Text-to-Image Generationcited by 3
8 more
  1. 2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
  2. 2026ROSE: Retrieval-Oriented Segmentation Enhancementcited by 3
  3. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  4. 2026MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memescited by 1
  5. 2025Bringing The Consistency Gap: Explicit Structured Memory for Interleaved Image-Text Generationcited by 0
  6. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  7. 2026GOPAgen: Codec-Aware Agentic Long-Video Understanding with Structured Memorycited by 0
  8. 2026SocialBuddy: Tailoring Search Agent for Social Scenarioscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  2. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
  3. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  4. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  5. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation
  6. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  7. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Models
  8. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning