mlsanityData to 14 Oct 2026

Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG

arXiv 2606.21968 ↗First posted 20 Jun 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Vision-Language Models (VLMs) struggle as query-relevant objects become smaller. To address this, recent training-free approaches dynamically retrieve and zoom into local image regions. However, we show that indiscriminately applying retrieval ignores a critical vulnerability: the resolution-context trade-off. Patch-based zooming recovers details for small targets, but can split large objects and destroy global spatial context; attention-based retrieval better preserves large objects, but remains less reliable on tiny details; and global perception is often fastest when retrieval is unnecessary. Motivated by these failure modes, we introduce ViRGo (Visual Retrieval or Global Perception), a lightweight framework that formulates visual retrieval as an adaptive routing problem. ViRGo estimates object scale from the VLM's intrinsic localization heads during the initial forward pass and combines it with semantic token confidence to select between global perception, patch-based retrieval, and attention-based retrieval with minimal additional computation. Experiments across multiple VQA benchmarks and object-size groups show that ViRGo improves the accuracy-efficiency trade-off: it matches patch retrieval on small details, leverages attention-based retrieval for larger objects, and reduces inference time by routing to the global baseline when zooming is unnecessary.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 25 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

15 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG3 Mar
  2. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs21 Dec

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  6. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  7. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  8. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  9. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 134
  10. 2021WebQA: Multihop and Multimodal QAcited by 109
  11. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  12. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
1 more
  1. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
  2. 2026Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
  3. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
  4. 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
  5. 2026Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
  6. 2026BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception
  7. 2025HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
  8. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval