mlsanityData to 7 Oct 2026

SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information

arXiv 2409.14083 ↗First posted 21 Sep 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

35 of 49 references.

References · 35

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
23 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  6. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  7. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  8. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  10. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  11. 2020Captioning Images Taken by People Who Are Blindcited by 108
  12. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  13. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 81
  14. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  15. 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
  16. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  17. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  18. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  19. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  20. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  21. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  22. 2022Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Changecited by 18
  23. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 9.

11 papers.

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  3. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  4. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Modelscited by 41
  7. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  8. 2025A Multimodal Multi-Agent Framework for Radiology Report Generationcited by 8
  9. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
  10. 2025Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approachcited by 0
  11. 2025LVLM-Aware Multimodal Retrieval for RAG-Based Medical Diagnosis with General-Purpose Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  2. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  3. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  4. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
  5. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  6. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  7. 2025Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
  8. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies