mlsanityData to 14 Oct 2026

Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG

arXiv 2503.01222 ↗First posted 3 Mar 2025cs.CV cs.CLcited by 23

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs). To overcome the limitations of existing methods, this paper shifts away from prior dedicated heuristic approaches and revisits the most fundamental idea to HR perception by enhancing the long-context capability of MLLMs, driven by recent advances in long-context techniques like retrieval-augmented generation (RAG) for general LLMs. Towards this end, this paper presents the first study exploring the use of RAG to address HR perception challenges. Specifically, we propose Retrieval-Augmented Perception (RAP), a training-free framework that retrieves and fuses relevant image crops while preserving spatial context using the proposed Spatial-Awareness Layout. To accommodate different tasks, the proposed Retrieved-Exploration Search (RE-Search) dynamically selects the optimal number of crops based on model confidence and retrieval scores. Experimental results on HR benchmarks demonstrate the significant effectiveness of RAP, with LLaVA-v1.5-13B achieving a 43% improvement on V^* Bench and 19% on HR-Bench.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 47 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

40 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs21 Dec

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Segment Anythingcited by 8,234
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
27 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023A Survey on Multimodal Large Language Modelscited by 909
  3. 2024Yi: Open Foundation Models by 01.AIcited by 719
  4. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  5. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  6. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  8. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  9. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  10. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  11. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 134
  12. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  13. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 113
  14. 2021WebQA: Multihop and Multimodal QAcited by 109
  15. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  16. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  17. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  18. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  19. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 60
  20. 2024Towards Robust Multimodal Sentiment Analysis with Incomplete Datacited by 27
  21. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  22. 2021Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networkscited by 25
  23. 2023Deep Graph Reprogrammingcited by 22
  24. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  25. 2022Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrievalcited by 19
  26. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  27. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 15.

23 papers.

Sort:

By area:

  1. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11
  2. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  3. 2026What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoningcited by 7
  4. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenderscited by 5
  5. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  6. 2025Wireless Agentic AI with Retrieval-Augmented Multimodal Semantic Perceptioncited by 4
  7. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  8. 2026The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Designcited by 3
  9. 2025Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspectivecited by 2
  10. 2025MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understandingcited by 1
  11. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  12. 2026Self-Prophetic Decoding to Unlock Visual Search in LVLMscited by 1
11 more
  1. 2026ActiveScope: Actively Seeking and Correcting Perception for MLLMscited by 1
  2. 2025Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformationcited by 0
  3. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  4. 2026Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Groundingcited by 0
  5. 2026Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAGcited by 0
  6. 2026Towards High-Resolution Visual Perception via Hierarchical Entity Explorationcited by 0
  7. 2026BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perceptioncited by 0
  8. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0
  9. 2026Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognitioncited by 0
  10. 2026The Earth in One Gaze: Training-Free Active Focus for UHR Remote Sensing Understandingcited by 0
  11. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook
  2. 2026CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
  3. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  4. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  5. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  6. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
  7. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  8. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance