mlsanityData to 7 Oct 2026

V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

arXiv 2312.14135 ↗First posted 21 Dec 2023cs.CVcited by 459

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.

Key facts. Cited by 459 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 60 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

44 of 60 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
32 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  13. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  15. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  16. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  17. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  18. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  19. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  20. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  21. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  22. 2023PACO: Parts and Attributes of Common Objectscited by 146
  23. 2019Zoom To Learn, Learn To Zoomcited by 114
  24. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  25. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  26. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  27. 2020Predicting Goal-directed Human Attention Using Inverse Reinforcement Learningcited by 53
  28. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 45
  29. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  30. 2018Finding any Waldo: zero-shot invariant and efficient visual searchcited by 28
  31. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8
  32. 2020Modeling human visual search: A combined Bayesian searcher and saliency map approach for eye movement guidance in natural scenescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 69; 2025: 162; 2026: 228.

459 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  9. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  10. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  11. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  12. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
38 more
  1. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  4. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  5. 2025Thyme: Think Beyond Imagescited by 160
  6. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  7. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  8. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  9. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  10. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  11. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  12. 2025MiMo-VL Technical Reportcited by 106
  13. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  14. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  15. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 94
  16. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  17. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  18. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  19. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  20. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  21. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  22. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  23. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  24. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  25. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  26. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  27. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  28. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  29. 2024When Do We Not Need Larger Vision Models?cited by 65
  30. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 64
  31. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  32. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  33. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  34. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  35. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  36. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  37. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  38. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47

50 most cited of 459 All 459 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
  2. 2026CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
  3. 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
  4. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
  5. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
  6. 2025InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
  7. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  8. 2026POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management